英文:

Integrating custom pytorch backend with triton + AWS sagemaker

问题

I have a custom python backend that works well with AWS sagemaker MMS (multimodel server) using an S3 model repository. I want to adapt this backend to work with Triton python backend.

我有一个自定义的Python后端，它在AWS SageMaker MMS（多模型服务器）上使用S3模型存储库运行得很好。我想将这个后端适配到Triton Python后端。

I have a example dockerfile that runs the triton server with my requirements.

我有一个示例的Dockerfile，它使用我的要求运行Triton服务器。

I also have a model_handler.py file that is based on this example, but I do not understand where to place this file to test its functionality. Using classic sagemaker with MMS for example, I would import the handler in the dockerd-entrypoint.

我还有一个基于这个示例的model_handler.py文件，但我不明白要将这个文件放在哪里来测试它的功能。例如，使用传统的SageMaker和MMS，我会在dockerd-entrypoint中导入handler。

However with triton, I do not understand where this file should be imported. I understand I can use pytriton, but there is absolutely no documentation that I can comprehend. Can someone point me in the right direction please?

然而，在Triton中，我不明白应该在哪里导入这个文件。我了解我可以使用pytriton，但没有我能理解的文档。请问有人可以指导我正确的方向吗？

英文:

I have a custom python backend that works well with AWS sagemaker MMS (multimodel server) using an S3 model repository. I want to adapt this backend to work with Triton python backend.
I have a example dockerfile that runs the triton server with my requirements.

I also have a model_handler.py file that is based on this example, but I do not understand where to place this file to test it's functionality. Using classic sagemaker with MMS for example, I would import the handler in the dockerd-entrypoint.

However with triton, I do not understand where this file should be imported. I understand I can use pytriton, but there is absolutely no documentation that I can comprehend. Can someone point me in the right direction please?

答案1

得分: 1

For Triton, 一个自定义推理脚本应该以 model.py 文件的形式提供。这个 model.py 实现了 initialize 方法（模型加载）、execute 方法和 finalize 方法，您可以在其中实现预处理和后处理逻辑。对于自定义的 Python 后端，您可以使用 conda-pack 定义环境并安装任何依赖项。在您的 config.pbtxt 文件中，您可以指向您定义和创建的这个环境。示例：https://github.com/aws/amazon-sagemaker-examples/tree/main/inference/nlp/realtime/triton/single-model/t5_pytorch_python-backend

英文:

For Triton a custom inference script is expected in the form of a model.py file. This model.py implements the initialize method (model loading), execute, and finalize methods where you can implement your pre/post processing logic. For the custom python backend you can install any additional dependencies by using conda-pack to define the environment and install any dependencies. In your config.pbtxt you can point towards this environment you have defined and created. Example: https://github.com/aws/amazon-sagemaker-examples/tree/main/inference/nlp/realtime/triton/single-model/t5_pytorch_python-backend