How can I save a model from a Sagemaker Pipelines TrainingStep in a specific location i.e. without the unique parent folder?

huangapple go评论58阅读模式
英文:

How can I save a model from a Sagemaker Pipelines TrainingStep in a specific location i.e. without the unique parent folder?

问题

我知道TrainingStep默认会将模型保存起来,但我想将其保存到我的S3存储桶的特定位置。我需要一种以编程方式找到模型存储位置的方法,因此我想摆脱唯一的父目录(pipelines-1913hs62gbn9-pipeline-scripts-tra-SAPd32cbD1z/)。

我知道有输出路径参数,但这只会更改唯一文件夹名称之后的路径,而不是整个路径。

我也知道这个问题以前已经得到了答复,但对我的情况没有帮助。(之前曾经受过伤害,所以现在在提问之前正在进行仔细的搜索)

例如
当前保存的路径:
s3://{my_bucket}/model/pipelines-1913hs62gbn9-pipeline-scripts-tra-SAPd32cbD1z/model.tar.gz

我想要的路径:
s3://{my_bucket}/model/model.tar.gz

有没有关于API调用或以特定方式保存的想法?

(注意:这只是原文的翻译,不包含代码部分。)

英文:

I know that the TrainingStep saves the model as output by default, but I want to save it in a specific place in my S3 bucket. I need a way to programmatically find where a model is stored, so I want to get rid of the unique parent directory (pipelines-1913hs62gbn9-pipeline-scripts-tra-SAPd32cbD1z/).

I am aware of the output path parameter, but this alters the path after the unique folder name, not the whole one.

I am also aware of this question which was previously answered, but is not helpful for my case. (Been burned before, so now I am doing a diligent search before asking)

For example
Path where it is saved now:
s3://{my_bucket}/model/pipelines-1913hs62gbn9-pipeline-scripts-tra-SAPd32cbD1z/model.tar.gz

Where I want it:
s3://{my_bucket}/model/model.tar.gz

Any ideas on API calls to make or ways to save it in a specific place?

*edited to fix spelling mistakes

答案1

得分: 1

SageMaker训练/流水线步骤总是为模型目录中存储的文件提供的输出路径添加唯一后缀。有两种方法可以避免这种情况:

  1. 不要将文件存储到/opt/ml/model,而是可以定义checkpoint_local_path和checkpoint_s3_uri,文件将按原样复制到S3。
  2. 编写自己的S3上传程序,将文件上传到已知路径。
英文:

SageMaker training/ pipeline step always adds a unique suffix to the output path provided for the files stored in the model dir. There are 2 ways you can avoid this

  1. Rather than storing to /opt/ml/model you can define checkpoint_local_path and checkpoint_s3_uri where the files are copied as is to s3.
  2. Write your own s3 uploader which will upload to a known path.

huangapple
  • 本文由 发表于 2023年6月2日 06:01:39
  • 转载请务必保留本文链接:https://go.coder-hub.com/76385996.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定