mlr3的类别权重是否应用于验证分数计算?

huangapple go评论65阅读模式
英文:

Are mlr3 class weights applied to validation score calculations?

问题

我之前使用过 mlr3 处理不平衡分类问题,并使用 PipeOpClassWeights 在训练过程中应用类别权重。这个管道操作会将观测权重列添加到 Task 中的 Task$weights 属性中。这些观测权重然后在训练期间传递给 Learner

一些分类性能指标,比如 Brier 得分(classif.bbrier)和对数损失(classif.logloss),可以在应用类别权重的情况下计算(这是在使用类别权重进行训练时发生的情况)。

我的问题是,在使用 resample 进行验证并从结果中聚合性能指标时,如下所示:

resampling = rsmp("cv", folds = 4)
measure = msr("classif.bbrier")
result = resample(task, learner, resampling)
score = resampling$aggregate(measure)

在这个验证得分计算中,类别权重是否也会被应用(如果适用于特定的度量标准)?在进行超参数调整时,例如使用 AutoTuner 时是否也会这样做?

我在上述类的文档和 mlr3book 的 resampling 部分 中查找了答案,但未找到答案。我认为我们希望将应用于训练损失的相同类别权重也应用于验证损失,至少对于超参数调整,如果不是性能测试的话。我在阅读到 xgboost 的 Python 实现时遇到了类似问题,可以在这里讨论。

英文:

I have previously used mlr3 for imbalanced classification problems, and used PipeOpClassWeights to apply class weights to learners during training. This pipe op adds a column of observation weights to the Task, in the Task$weights property. These observation weights are then passed to the Learner during training.

Some classification performance metrics, such as Brier score (classif.bbrier) and log loss (classif.logloss) can be calculated with class weights applied (this is what happens to the training log loss when we train with class weights).

My question is, when we perform validation with resample, and aggregate performance metrics from the results, in the form of

resampling = rsmp("cv", folds = 4)
measure = msr("classif.bbrier")
result = resample(task, learner, resampling)
score = resampling$aggregate(measure)

are the class weights applied to this validation score calculation too (if applicable to the particular measure)? Is this also done when we perform hyperparameter tuning, for example with AutoTuner?

I looked in the documentation for the aforementioned classes, and the resampling section of mlr3book, but couldn't find an answer. I assume we'd want the same class weights applied to the training loss to be applied to the validation loss, at least for hyperparameter tuning if not for performance testing.

I was inspired to investigate this after coming across a similar issue in the validation scores of xgboost's Python implementation, discussed here.

答案1

得分: 0

是的,它们是的。如果它们不是的话,那就是一个错误。

英文:

Yes, they are. If they're not, that's a bug.

huangapple
  • 本文由 发表于 2023年5月17日 20:18:03
  • 转载请务必保留本文链接:https://go.coder-hub.com/76272021.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定