需要GPU（cuda）访问在部署模型时

huangapple
go评论101阅读模式
英文:

Need GPU (cuda) access while deploying the model

问题

I need assistance with deploying a pre-trained model. I have created a custom score.py file for the deployment process. However, the docker created on the CPU instance does not provide access to the GPU, which poses a problem for predicting with PyTorch or TensorFlow models as they require input to be converted to tensors loaded on the GPU. Can you suggest a solution?

My score.py script -

  1. import something
  3. def init():
  4.     global model
  6.     model_path = os.path.join(os.getenv("AZUREML_MODEL_DIR"), "use-case1-model")
  8.     model = mlflow.pytorch.load_model(model_path, map_location=torch.device('cpu'))
  9.     logging.info("Init complete")
  11. tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
  13. def run(data):
  15.     json_data = json.loads(data) 
  17.     title = json_data["input_data"]["title"]
  18.     att = json_data["input_data"]["attributes"]
  19.     
  20.     result = {}
  22.     for i in range(len(title)):
  24.         my_dict = {}
  25.         for j in range(len(att)):
  26.             
  27.             attr = att[i][j]
  29.             t, a = nobert4token(tokenizer, title[i].lower(), attr)
  31.             x = X_padding(t)
  32.             y = tag_padding(a)
  34.             tensor_a = torch.tensor(y, dtype=torch.int32)
  35.             tensor_a = torch.unsqueeze(tensor_a, dim=0).to("cuda")
  37.             tensor_t = torch.tensor(x, dtype=torch.int32)
  38.             tensor_t = torch.unsqueeze(tensor_t, dim=0).to("cuda")
  40.             output = model([tensor_t, tensor_a])
  42.             predict_list = output.tolist()[0]
  43.             
  44.             my_dict[attr] = " ".join(words_p)
  46.         result[title[i]] = my_dict
  48.     return result

My invoke script-

  1. ml_client.online_endpoints.invoke(
  2.     endpoint_name=endpoint_result.name,
  3.     deployment_name=green_deployment_uc1.name,
  4.     request_file=os.path.join("./dependencies", "sample.json"),
  5. )

My conda.yaml-

  1. channels:
  2.   - conda-forge
  3. dependencies:
  4.   - python=3.8
  5.   - pip=22.1.2
  6.   - numpy=1.21.2
  7.   - scikit-learn=0.24.2
  8.   - scipy=1.7.1
  9.   - 'pandas>=1.1,<1.2'
  10.   - pytorch=1.10.0
  11.   - pip:
  12.       - 'inference-schema[numpy-support]==1.5.0'
  13.       - xlrd==2.0.1
  14.       - mlflow==1.26.1
  15.       - azureml-mlflow==1.42.0
  16.       - tqdm==4.63.0
  17.       - pytorch-transformers==1.2.0
  18.       - pytorch-lightning==2.0.2
  19.       - seqeval==1.2.2
  20.       - azureml-inference-server-http==0.8.0
  21. name: model-env

Error that I am getting -

  1. 127.0.0.1 - - [29/May/2023:10:03:32 +0000] "GET / HTTP/1.0" 200 7 "-" "kube-probe/1.18"
  2. 2023-05-29 10:03:34,291 E [70] azmlinfsrv - Encountered Exception: Traceback (most recent call last):
  3.   File "/azureml-envs/azureml_d587e0800be72e17d773ddca63762cd1/lib/python3.8/site-packages/azureml_inference_server_http/server/user_script.py", line 130, in invoke_run
  4.     run_output = self._wrapped_user_run(**run_parameters, request_headers=dict(request.headers))
  5.   File "/azureml-envs/azureml_d587e0800be72e17d773ddca63762cd1/lib/python3.8/site-packages/azureml_inference_server_http/server/user_script.py", line 154, in <lambda>
  6.     self._wrapped_user_run = lambda request_headers, **kwargs: self._user_run(**kwargs)
  7.   File "/var/azureml-app/dependencies/score.py", line 129, in run
  8.     tensor_a = torch.unsqueeze(tensor_a, dim=0).to("cuda")
  9.   File "/azureml-envs/azureml_d587e0800be72e17d773ddca63762cd1/lib/python3.8/site-packages/torch/cuda/__init__.py", line 247, in _lazy_init
  10.     torch._C._cuda_init()
  11. RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx
  13. The above exception was the direct cause of the following exception:

If you think why I used "model = mlflow.pytorch.load_model(model_path, map_location=torch.device('cpu'))"

please refer to this forum- https://learn.microsoft.com/en-us/answers/questions/1291498/facing-problem-while-deploying-model-on-azure-ml-a

Documentation - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-models-online-endpoints?view=azureml-api-2&tabs=sdk

英文:

I need assistance with deploying a pre-trained model. I have created a custom score.py file for the deployment process. However, the docker created on the CPU instance does not provide access to the GPU, which poses a problem for predicting with PyTorch or TensorFlow models as they require input to be converted to tensors loaded on the GPU. Can you suggest a solution?

My score.py script -

  1. import something
  3. # original = torch.load
  6. # def load(*args):
  7. #     return torch.load(*args, map_location=torch.device(&quot;cpu&quot;),pickle_module=None)
  10. # def init():
  11. #     global model
  12. #     model_path = os.path.join(os.getenv(&quot;AZUREML_MODEL_DIR&quot;), &quot;use-case1-model&quot;)
  13. #     # &quot;model&quot; is the path of the mlflow artifacts when the model was registered. For automl
  14. #     # models, this is generally &quot;mlflow-model&quot;.
  16. #     with mock.patch(&quot;torch.load&quot;, load):
  17. #         model = mlflow.pyfunc.load_model(model_path)
  19. #     logging.info(&quot;Init complete&quot;)
  21. def init():
  22.     global model
  24.     model_path = os.path.join(os.getenv(&quot;AZUREML_MODEL_DIR&quot;), &quot;use-case1-model&quot;)
  26.     model = mlflow.pytorch.load_model(model_path, map_location=torch.device(&#39;cpu&#39;))
  27.     logging.info(&quot;Init complete&quot;)
  30. tokenizer = BertTokenizer.from_pretrained(&quot;bert-base-uncased&quot;)
  31.             
  33. def run(data):
  35.     json_data = json.loads(data) 
  37.     title = json_data[&quot;input_data&quot;][&quot;title&quot;]
  38.     att = json_data[&quot;input_data&quot;][&quot;attributes&quot;]
  39.     
  40.     result = {}
  42.     for i in range(len(title)):
  44.         my_dict = {}
  45.         for j in range(len(att)):
  46.             
  47.             attr = att[i][j]
  49.             t, a = nobert4token(tokenizer, title[i].lower(), attr)
  51.             x = X_padding(t)
  52.             y = tag_padding(a)
  54.             tensor_a = torch.tensor(y, dtype=torch.int32)
  55.             tensor_a = torch.unsqueeze(tensor_a, dim=0).to(&quot;cuda&quot;)
  57.             tensor_t = torch.tensor(x, dtype=torch.int32)
  58.             tensor_t = torch.unsqueeze(tensor_t, dim=0).to(&quot;cuda&quot;)
  60.             output = model([tensor_t, tensor_a])
  62.             predict_list = output.tolist()[0]
  63.             
  64.             my_dict[attr] = &quot; &quot;.join(words_p)
  66.         result[title[i]] = my_dict
  69.     return result

My invoke script-

  1. ml_client.online_endpoints.invoke(
  2.     endpoint_name=endpoint_result.name,
  3.     deployment_name=green_deployment_uc1.name,
  4.     request_file=os.path.join(&quot;./dependencies&quot;, &quot;sample.json&quot;),
  5. )

My conda.yaml-

  1. channels:
  2.   - conda-forge
  3. dependencies:
  4.   - python=3.8
  5.   - pip=22.1.2
  6.   - numpy=1.21.2
  7.   - scikit-learn=0.24.2
  8.   - scipy=1.7.1
  9.   - &#39;pandas&gt;=1.1,&lt;1.2&#39;
  10.   - pytorch=1.10.0
  11.   - pip:
  12.       - &#39;inference-schema[numpy-support]==1.5.0&#39;
  13.       - xlrd==2.0.1
  14.       - mlflow== 1.26.1
  15.       - azureml-mlflow==1.42.0
  16.       - tqdm==4.63.0
  17.       - pytorch-transformers==1.2.0
  18.       - pytorch-lightning==2.0.2
  19.       - seqeval==1.2.2
  20.       - azureml-inference-server-http==0.8.0
  21. name: model-env

Error that I am getting -

  1. 127.0.0.1 - - [29/May/2023:10:03:32 +0000] &quot;GET / HTTP/1.0&quot; 200 7 &quot;-&quot; &quot;kube-probe/1.18&quot;
  2. 2023-05-29 10:03:34,291 E [70] azmlinfsrv - Encountered Exception: Traceback (most recent call last):
  3.   File &quot;/azureml-envs/azureml_d587e0800be72e17d773ddca63762cd1/lib/python3.8/site-packages/azureml_inference_server_http/server/user_script.py&quot;, line 130, in invoke_run
  4.     run_output = self._wrapped_user_run(**run_parameters, request_headers=dict(request.headers))
  5.   File &quot;/azureml-envs/azureml_d587e0800be72e17d773ddca63762cd1/lib/python3.8/site-packages/azureml_inference_server_http/server/user_script.py&quot;, line 154, in &lt;lambda&gt;
  6.     self._wrapped_user_run = lambda request_headers, **kwargs: self._user_run(**kwargs)
  7.   File &quot;/var/azureml-app/dependencies/score.py&quot;, line 129, in run
  8.     tensor_a = torch.unsqueeze(tensor_a, dim=0).to(&quot;cuda&quot;)
  9.   File &quot;/azureml-envs/azureml_d587e0800be72e17d773ddca63762cd1/lib/python3.8/site-packages/torch/cuda/__init__.py&quot;, line 247, in _lazy_init
  10.     torch._C._cuda_init()
  11. RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx
  13. The above exception was the direct cause of the following exception:

If you think why I used "model = mlflow.pytorch.load_model(model_path, map_location=torch.device('cpu'))"

please refer to this forum- https://learn.microsoft.com/en-us/answers/questions/1291498/facing-problem-while-deploying-model-on-azure-ml-a

Documentation - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-models-online-endpoints?view=azureml-api-2&amp;tabs=sdk

答案1

得分: 0

为了解决这个问题，您可以修改您的代码，确保张量加载到CPU而不是GPU上。
在您的代码中添加设备变量：

  1. device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

在**run()**函数中替换以下代码：

  1. tensor_a = torch.tensor(y, dtype=torch.int32)
  2. tensor_a = torch.unsqueeze(tensor_a, dim=0).to(device)
  4. tensor_t = torch.tensor(x, dtype=torch.int32)
  5. tensor_t = torch.unsqueeze(tensor_t, dim=
  7. <details>
  8. <summary>英文:</summary>
  10. To solve this issue, you can modify your code to ensure that the tensors are loaded onto the CPU instead of the GPU.
  11. Add device variable in your code: 
  13.     device = torch.device(&quot;cuda&quot;  if  torch.cuda.is_available() else  &quot;cpu&quot;)
  15. Replace below code in **run ()** function:
  17.      tensor_a = torch.tensor(y, dtype=torch.int32)
  18.      tensor_a = torch.unsqueeze(tensor_a, dim=0).to(&quot;device&quot;)
  19.     
  20.      tensor_t = torch.tensor(x, dtype=torch.int32)
  21.      tensor_t = torch.unsqueeze(tensor_t, dim=0).to(&quot;device&quot;)
  23.   Below is the example for the error and fix.
  25. Error Reproduced;
  26. [![enter image description here][1]][1]
  28. Fix:
  30. [![enter image description here][2]][2]
  33.   [1]: https://i.stack.imgur.com/kSi5E.png
  34.   [2]: https://i.stack.imgur.com/CzwnO.png
  36. </details>

huangapple
  • 本文由 发表于 2023年5月29日 18:44:30
  • 转载请务必保留本文链接：https://go.coder-hub.com/76356663.html
  • azure
  • azure-deployment
  • azure-machine-learning-service
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定