英文:
Are mlr3 class weights applied to validation score calculations?
问题
我之前使用过 mlr3
处理不平衡分类问题,并使用 PipeOpClassWeights
在训练过程中应用类别权重。这个管道操作会将观测权重列添加到 Task
中的 Task$weights
属性中。这些观测权重然后在训练期间传递给 Learner
。
一些分类性能指标,比如 Brier 得分(classif.bbrier
)和对数损失(classif.logloss
),可以在应用类别权重的情况下计算(这是在使用类别权重进行训练时发生的情况)。
我的问题是,在使用 resample
进行验证并从结果中聚合性能指标时,如下所示:
resampling = rsmp("cv", folds = 4)
measure = msr("classif.bbrier")
result = resample(task, learner, resampling)
score = resampling$aggregate(measure)
在这个验证得分计算中,类别权重是否也会被应用(如果适用于特定的度量标准)?在进行超参数调整时,例如使用 AutoTuner
时是否也会这样做?
我在上述类的文档和 mlr3book 的 resampling 部分 中查找了答案,但未找到答案。我认为我们希望将应用于训练损失的相同类别权重也应用于验证损失,至少对于超参数调整,如果不是性能测试的话。我在阅读到 xgboost
的 Python 实现时遇到了类似问题,可以在这里讨论。
英文:
I have previously used mlr3
for imbalanced classification problems, and used PipeOpClassWeights
to apply class weights to learners during training. This pipe op adds a column of observation weights to the Task
, in the Task$weights
property. These observation weights are then passed to the Learner
during training.
Some classification performance metrics, such as Brier score (classif.bbrier
) and log loss (classif.logloss
) can be calculated with class weights applied (this is what happens to the training log loss when we train with class weights).
My question is, when we perform validation with resample
, and aggregate performance metrics from the results, in the form of
resampling = rsmp("cv", folds = 4)
measure = msr("classif.bbrier")
result = resample(task, learner, resampling)
score = resampling$aggregate(measure)
are the class weights applied to this validation score calculation too (if applicable to the particular measure)? Is this also done when we perform hyperparameter tuning, for example with AutoTuner
?
I looked in the documentation for the aforementioned classes, and the resampling section of mlr3book, but couldn't find an answer. I assume we'd want the same class weights applied to the training loss to be applied to the validation loss, at least for hyperparameter tuning if not for performance testing.
I was inspired to investigate this after coming across a similar issue in the validation scores of xgboost
's Python implementation, discussed here.
答案1
得分: 0
是的,它们是的。如果它们不是的话,那就是一个错误。
英文:
Yes, they are. If they're not, that's a bug.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论