如何在Optuna中记录交叉验证中每个折叠的验证损失？

huangapple
go评论140阅读模式
英文:

How to record each fold`s validation loss during cross-validation in Optuna?

问题

抱歉，我只会为您提供代码的翻译。以下是您提供的代码的翻译：

  1. 我正在使用[Toshihiko Yanase](https://stackoverflow.com/a/63242079/21220510)的代码来执行我的超参数优化器的交叉验证，使用Optuna。以下是我正在使用的代码：
  3. def objective(trial, train_loader, valid_loader):
  5.     # 移除以下行。
  6.     # train_loader, valid_loader = get_mnist()
  8.     ...
  10.     return accuracy
  13. def objective_cv(trial):
  15.     # 获取MNIST数据集。
  16.     dataset = datasets.MNIST(DIR, train=True, download=True, transform=transforms.ToTensor())
  18.     fold = KFold(n_splits=3, shuffle=True, random_state=0)
  19.     scores = []
  20.     for fold_idx, (train_idx, valid_idx) in enumerate(fold.split(range(len(dataset))):
  21.         train_data = torch.utils.data.Subset(dataset, train_idx)
  22.         valid_data = torch.utils.data.Subset(dataset, valid_idx)
  24.         train_loader = torch.utils.data.DataLoader(
  25.             train_data,
  26.             batch_size=BATCHSIZE,
  27.             shuffle=True,
  28.         )
  29.         valid_loader = torch.utils.data.DataLoader(
  30.             valid_data,
  31.             batch_size=BATCHSIZE,
  32.             shuffle=True,
  33.         )
  35.         accuracy = objective(trial, train_loader, valid_loader)
  36.         scores.append(accuracy)
  37.     return np.mean(scores)
  40. study = optuna.create_study(direction="maximize")
  41. study.optimize(objective_cv, n_trials=20, timeout=600)

很抱歉，您的问题涉及非代码部分，我无法回答。如果您有任何其他需要翻译的代码，请随时提问。

英文:

I am using Toshihiko Yanase`s code for doing cross validation on my hyperparameter optimizer with Optuna. Here is the code that I am using:

  1. def objective(trial, train_loader, valid_loader):
  3.     # Remove the following line.
  4.     # train_loader, valid_loader = get_mnist()
  6.     ...
  8.     return accuracy
  11. def objective_cv(trial):
  13.     # Get the MNIST dataset.
  14.     dataset = datasets.MNIST(DIR, train=True, download=True, transform=transforms.ToTensor())
  16.     fold = KFold(n_splits=3, shuffle=True, random_state=0)
  17.     scores = []
  18.     for fold_idx, (train_idx, valid_idx) in enumerate(fold.split(range(len(dataset)))):
  19.         train_data = torch.utils.data.Subset(dataset, train_idx)
  20.         valid_data = torch.utils.data.Subset(dataset, valid_idx)
  22.         train_loader = torch.utils.data.DataLoader(
  23.             train_data,
  24.             batch_size=BATCHSIZE,
  25.             shuffle=True,
  26.         )
  27.         valid_loader = torch.utils.data.DataLoader(
  28.             valid_data,
  29.             batch_size=BATCHSIZE,
  30.             shuffle=True,
  31.         )
  33.         accuracy = objective(trial, train_loader, valid_loader)
  34.         scores.append(accuracy)
  35.     return np.mean(scores)
  38. study = optuna.create_study(direction="maximize")
  39. study.optimize(objective_cv, n_trials=20, timeout=600)

Unfortunately, using the code this way, it does not record each folds val loss to the Optuna dashboard. Is there a way to record each folds val loss to the Optuna dashboard?

答案1

得分: 1

每个拆分的验证损失可以记录在当前试验的Trial对象的system_attrs中。system_attrs可以在仪表板中看到，就像您所希望的那样。

具有所需功能的修改后代码如下：

  1. def objective(trial, train_loader, valid_loader):
  2.     
  3.     # 移除以下行。
  4.     # train_loader, valid_loader = get_mnist()
  5.     
  6.     ...
  7.     
  8.     return accuracy
  10. def objective_cv(trial):
  11.     
  12.     # 获取MNIST数据集。
  13.     dataset = datasets.MNIST(DIR, train=True, download=True, transform=transforms.ToTensor())
  15.     fold = KFold(n_splits=3, shuffle=True, random_state=0)
  16.     scores = []
  17.     trial.set_system_attr("Val loss of fold", [])  # 用于记录当前折的每个个体最终损失
  18.     for fold_idx, (train_idx, valid_idx) in enumerate(fold.split(range(len(dataset))):
  19.         train_data = torch.utils.data.Subset(dataset, train_idx)
  20.         valid_data = torch.utils.data.Subset(dataset, valid_idx)
  22.         train_loader = torch.utils.data.DataLoader(
  23.             train_data,
  24.             batch_size=BATCHSIZE,
  25.             shuffle=True,
  26.         )
  27.         valid_loader = torch.utils.data.DataLoader(
  28.             valid_data,
  29.             batch_size=BATCHSIZE,
  30.             shuffle=True,
  31.         )
  33.         accuracy = objective(trial, train_loader, valid_loader)
  34.         scores.append(accuracy)
  35.         trial.set_system_attr("Val loss of fold", trial.system_attrs["Val loss of fold"] + [accuracy])  # 这里添加了目标值以记录
  36.     return np.mean(scores)
  38. study = optuna.create_study(direction="maximize")
  39. study.optimize(objective_cv, n_trials=20, timeout=600)
  41. [PS：不幸的是，Optuna的开发人员已经表示他们将在未来删除`system_attrs`，我认为这将是一个损失。](2)
  43. [1]: https://optuna.readthedocs.io/en/stable/reference/generated/optuna.trial.Trial.html#optuna.trial.Trial:~:text=to%20be%20optimized.-,system_attrs,-Return%20system%20attributes
  44. [2]: https://Deprecated%20in%20v3.1.0.%20This%20feature%20will%20be%20removed%20in%20the%20future.%20The%20removal%20of%20this%20feature%20is%20currently%20scheduled%20for%20v6.0.0,%20but%20this%20schedule%20is%20subject%20to%20change.%20See%20https://github.com/optuna/optuna/releases/tag/v3.1.0.
  46. <details>
  47. <summary>英文:</summary>
  49. Each splits validation loss can be recorded in the [`system_attrs`][1] of the Trial object of the current trial. The `system_attrs` can be seen in the dashboard under the respective trial as you wished.
  51. The modified code having the desired functionality is:
  53.     def objective(trial, train_loader, valid_loader):
  54.     
  55.         # Remove the following line.
  56.         # train_loader, valid_loader = get_mnist()
  57.     
  58.         ...
  59.     
  60.         return accuracy
  61.     
  62.     
  63.     def objective_cv(trial):
  64.     
  65.         # Get the MNIST dataset.
  66.         dataset = datasets.MNIST(DIR, train=True, download=True, transform=transforms.ToTensor())
  67.     
  68.         fold = KFold(n_splits=3, shuffle=True, random_state=0)
  69.         scores = []
  70.         trial.set_system_attr(&quot;Val loss of fold&quot;,[])   #to record each individual final loss of the current fold
  71.         for fold_idx, (train_idx, valid_idx) in enumerate(fold.split(range(len(dataset)))):
  72.             train_data = torch.utils.data.Subset(dataset, train_idx)
  73.             valid_data = torch.utils.data.Subset(dataset, valid_idx)
  74.     
  75.             train_loader = torch.utils.data.DataLoader(
  76.                 train_data,
  77.                 batch_size=BATCHSIZE,
  78.                 shuffle=True,
  79.             )
  80.             valid_loader = torch.utils.data.DataLoader(
  81.                 valid_data,
  82.                 batch_size=BATCHSIZE,
  83.                 shuffle=True,
  84.             )
  85.     
  86.             accuracy = objective(trial, train_loader, valid_loader)
  87.             scores.append(accuracy)
  88.             trial.set_system_attr(&quot;Val loss of fold&quot;,trial.system_attrs[&quot;Val loss of fold&quot;]+[accuracy]) #here is the objective value is added to the record
  89.         return np.mean(scores)
  90.     
  91.     
  92.     study = optuna.create_study(direction=&quot;maximize&quot;)
  93.     study.optimize(objective_cv, n_trials=20, timeout=600)
  95. [PS: Unfortunatly, Optuna developers have indicated that they will remove the `system_attrs` in the future which I think will be a loss.][2] 
  98.   [1]: https://optuna.readthedocs.io/en/stable/reference/generated/optuna.trial.Trial.html#optuna.trial.Trial:~:text=to%20be%20optimized.-,system_attrs,-Return%20system%20attributes
  99.   [2]: https://Deprecated%20in%20v3.1.0.%20This%20feature%20will%20be%20removed%20in%20the%20future.%20The%20removal%20of%20this%20feature%20is%20currently%20scheduled%20for%20v6.0.0,%20but%20this%20schedule%20is%20subject%20to%20change.%20See%20https://github.com/optuna/optuna/releases/tag/v3.1.0.
  101. </details>

huangapple
  • 本文由 发表于 2023年2月27日 06:14:56
  • 转载请务必保留本文链接：https://go.coder-hub.com/75575325.html
  • deep-learning
  • optuna
  • python
  • pytorch
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定