Cannot import protobuf builder when deploying Azure function

The Problem

I have an Azure function built using the Python v1 programming model which builds a basic Keras model and trains it using some data from an Azure file share.

Tested locally and everything works.

I'm deploying to Azure using Azure Pipelines. It's invoking correctly, but then failing. Checking the detailed invocation history reveals the following error:

Result: Failure Exception: ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/azure-functions-host/workers/python/3.9/LINUX/X64/google/protobuf/internal/__init__.py). 
Please check the requirements.txt file for the missing module. For more info, please refer the troubleshooting guide: https://aka.ms/functions-modulenotfound
Stack: File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 380, in _handle__function_load_request func = loader.load_function( 
File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 48, in call raise extend_exception_message(e, message) File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call return func(*args, **kwargs)
File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 132, in load_function mod = importlib.import_module(fullmodname)
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked 
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed 
File "/home/site/wwwroot/func-gc-imgorientation-train/__init__.py", line 20, in <module> from core.data.training_image_filestore import TrainingImageFilestore
File "/home/site/wwwroot/core/data/training_image_filestore.py", line 4, in <module> import tensorflow as tf
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/__init__.py", line 37, in <module> from tensorflow.python.tools import module_util as _module_util
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/python/__init__.py", line 37, in <module> from tensorflow.python.eager import context
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/python/eager/context.py", line 28, in <module> from tensorflow.core.framework import function_pb2
File "/home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/core/framework/function_pb2.py", line 5, in <module> from google.protobuf.internal import builder as _builder

Well Documented Error with a Solution that Works Outside of Azure

This error is well documented and understood as in the following version:

https://stackoverflow.com/questions/71759248/importerror-cannot-import-name-builder-from-google-protobuf-internal

In general, arising from API changes in v3.20:

https://stackoverflow.com/a/71984564/1928761

TF adopted these changes, but there were bugs when using TF with protobuf >= v3.20 which were resolved in the most recent Tensorflow release so protobuf 4.23.3 should work with TF 2.12.0 as here:

https://github.com/tensorflow/tensorflow/issues/59221

Problem is resolved locally

I resolved this issue locally and have tested my function. All works fine. The problem only occurs in Azure.

Suspected Root Cause in Azure

Looking through the error above I noticed that all my custom modules, and Tensorflow, are installed to the same root: /home/site/wwwroot/

For instance, my custom core.data module is installed at /home/site/wwwroot/core/data/

Tensorflow is installed at /home/site/wwwroot/.python_packages/lib/site-packages/tensorflow/

I've confirmed that all modules in my requirements.txt are being installed as expected in /home/site/wwwroot/.python_packages/lib/site-packages/ - including the latest protobuf with the builder module as expected.

However, protobuf is being imported from /azure-functions-host/workers/python/3.9/LINUX/X64/google/protobuf/internal/init.py

Presumably the implication is that TF is using the version of protobuf that's bundled with Azure's python distribution on the host, rather than the version in my site packages.

To test this theory I added the following code to the top of my init.py for the function:

import sys
print(sys.path)
import google.protobuf
print(google.protobuf.__version__)
print(google.protobuf.__path__)

This confirmed that protobuf version 3.19 was being loaded from the following path and not from site packages: /azure-functions-host/workers/python/3.9/LINUX/X64/google/protobuf/

The Question

The question therefore is can I upgrade the version of protobuf that's included in this python bundle?

Alternatively, can I force TF to use the version of protobuf in my site packages rather than the ones in the Azure python bundle?

Attempted Solutions

ADO is building and installing the correct versions of both protobuf and TF, and the builder file is definitely in the installed site-packages. I've confirmed this by downloading the package from my storage account and unzipping it.

To resolve the issue, I've tried the following:

1. Confirmed that I can import other modules and that this is specific to importing the protobuf builder
2. Confirmed that I can import google.protobuf.internal.
3. Tried clearing the protobuf pycache by adding an rm -rvf command to my bash script immediately after the pip install --target....
4. Using subprocess to run pip install --upgrade protobuf==4.23.3 from within my function and before importing TF (which runs without error, but does not do the job).

At this point, I'm all out of ideas.

requirements.txt

# DO NOT include azure-functions-worker in this file
# The Python Worker is managed by Azure Functions platform
# Manually managing azure-functions-worker may cause unexpected issues
# Protobuf comes first to force the very latest version
protobuf==4.23.3
# Tensorflow, Dotenv, Pillow
absl-py==1.4.0
astunparse==1.6.3
cachetools==5.3.1
certifi==2023.5.7
charset-normalizer==3.1.0
contourpy==1.1.0
cycler==0.11.0
flatbuffers==23.5.26
fonttools==4.40.0
gast==0.4.0
google-auth==2.20.0
google-auth-oauthlib==1.0.0
google-pasta==0.2.0
graphviz==0.20.1
grpcio==1.54.2
h5py==3.8.0
idna==3.4
jax==0.4.12
keras==2.12.0
kiwisolver==1.4.4
libclang==16.0.0
Markdown==3.4.3
MarkupSafe==2.1.3
matplotlib==3.7.1
ml-dtypes==0.2.0
numpy==1.23.5
oauthlib==3.2.2
opt-einsum==3.3.0
packaging==23.1
Pillow==9.5.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pydot==1.4.2
pyparsing==3.0.9
python-dateutil==2.8.2
python-dotenv
requests==2.31.0
requests-oauthlib==1.3.1
rsa==4.9
scipy==1.10.1
six==1.16.0
tensorboard==2.12.3
tensorboard-data-server==0.7.1
tensorflow==2.12.0
tensorflow-estimator==2.12.0
termcolor==2.3.0
typing_extensions==4.6.3
urllib3==1.26.16
Werkzeug==2.3.6
wrapt==1.14.1
# Azure libraries
azure-functions
azure-identity
azure-keyvault

Deployment pipeline YAML

# Python Function App to Linux on Azure
# Build a Python function app and deploy it to Azure as a Linux function app.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/python
trigger:
- main
variables:
# Azure Resource Manager connection created during pipeline creation
azureSubscription: '4ae24131-0b22-421c-8e3e-6d766e891ece'
# Function app name
functionAppName: 'func-xxxx-dev'
# Agent VM image name
vmImageName: 'ubuntu-latest'
# Working Directory
workingDirectory: '$(System.DefaultWorkingDirectory)'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- bash: |
if [ -f extensions.csproj ]
then
dotnet build extensions.csproj --runtime ubuntu.16.04-x64 --output ./bin
fi
workingDirectory: $(workingDirectory)
displayName: 'Build extensions'
- task: UsePythonVersion@0
displayName: 'Use Python 3.9'
inputs:
versionSpec: 3.9 # Functions V2 supports Python 3.6 as of today
- bash: |
python -m pip install --upgrade pip
pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
workingDirectory: $(workingDirectory)
displayName: 'Install application dependencies'
- task: ArchiveFiles@2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(workingDirectory)'
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- publish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
artifact: drop
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: Deploy
displayName: Deploy
environment: 'development'
pool:
vmImage: $(vmImageName)
strategy:
runOnce:
deploy:
steps:
- task: AzureFunctionApp@1
displayName: 'Azure functions app deploy'
inputs:
azureSubscription: '$(azureSubscription)'
appType: functionAppLinux
appName: $(functionAppName)
package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'

Log stream

2023-06-27T11:50:43Z   [Information]   Host Status: {
"id": "func-xxxx-dev",
"state": "Running",
"version": "4.21.3.3",
"versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
"platformVersion": "",
"instanceId": "9A4EF22A-638234606489642041",
"computerName": "",
"processUptime": 2778401,
"functionAppContentEditingState": "NotAllowed",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "4.5.0"
}
}
2023-06-27T11:50:43Z   [Information]   Host Status: {
"id": "func-xxxx-dev",
"state": "Running",
"version": "4.21.3.3",
"versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
"platformVersion": "",
"instanceId": "9A4EF22A-638234606489642041",
"computerName": "",
"processUptime": 2778404,
"functionAppContentEditingState": "NotAllowed",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "4.5.0"
}
}
2023-06-27T11:50:43Z   [Information]   Host Status: {
"id": "func-xxxx-dev",
"state": "Running",
"version": "4.21.3.3",
"versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
"platformVersion": "",
"instanceId": "9A4EF22A-638234606489642041",
"computerName": "",
"processUptime": 2778801,
"functionAppContentEditingState": "NotAllowed",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "4.5.0"
}
}
2023-06-27T11:51:00Z   [Information]   Executing 'Functions.func-xxxx-train' (Reason='Timer fired at 2023-06-27T11:51:00.0016150+00:00', Id=3795600b-379e-423c-b29f-65fec390289a)
2023-06-27T11:51:00Z   [Verbose]   Sending invocation id: '3795600b-379e-423c-b29f-65fec390289a
2023-06-27T11:51:00Z   [Verbose]   Posting invocation id:3795600b-379e-423c-b29f-65fec390289a on workerId:250ae2e1-4416-41ef-b355-0684a59d0a91
2023-06-27T11:51:00Z   [Error]   Executed 'Functions.func-xxxx-train' (Failed, Id=3795600b-379e-423c-b29f-65fec390289a, Duration=2ms)
2023-06-27T11:51:00Z   [Verbose]   Function 'func-xxx-train' updated status: Last='2023-06-27T11:51:00.0015724+00:00', Next='2023-06-27T11:52:00.0000000+00:00', LastUpdated='2023-06-27T11:51:00.0015724+00:00'
2023-06-27T11:51:00Z   [Verbose]   Timer for 'func-gc-imgorientation-train' started with interval '00:00:59.9674482'.
2023-06-27T11:51:05Z   [Information]   Host Status: {
"id": "func-gc-imgorientation-dev",
"state": "Running",
"version": "4.21.3.3",
"versionDetails": "4.21.3+2e42e3beb40b89d4f5d3dd962f3a5d420d376d71",
"platformVersion": "",
"instanceId": "9A4EF22A-638234606489642041",
"computerName": "",
"processUptime": 2800545,
"functionAppContentEditingState": "NotAllowed",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "4.5.0"
}
}
</details>
# 答案1
**得分**: 0
If you want to use Protobuf module in Azure Functions, You need to downgrade it to ***3.20.**** and add it in your requirements.txt.
我已将 protobuf==3.20.* 添加到我的 requirements.txt 中，Http 触发器通过 DevOps YAML 管道成功部署到 Azure Functions 中，参考以下内容:-
**我的 requirements.txt:-**
```txt
azure-functions
protobuf==3.20.*

我的 init.py:-

import logging

import azure.functions as func


def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get('name')

    if name:
        return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully.")
    else:
        return func.HttpResponse(
             "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
             status_code=200
        )

我的 YAML 管道:-

trigger:
- master

variables:
  
  azureSubscription: 'xxxxxxxx-xxxxx-xxx9bbd4354dd'

 
  functionAppName: 'valleyfunc541'

 
  vmImageName: 'ubuntu-latest'

 
  workingDirectory: '$(System.DefaultWorkingDirectory)'

stages:
- stage: Build
  displayName: Build stage

  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)

    steps:
    - bash: |
        if [ -f extensions.csproj ]
        then
            dotnet build extensions.csproj --runtime ubuntu.16.04-x64 --output ./bin
        fi        
      workingDirectory: $(workingDirectory)
      displayName: 'Build extensions'

    - task: UsePythonVersion@0
      displayName: 'Use Python 3.10'
      inputs:
        versionSpec: 3.10 # Functions V2 supports Python 3.6 as of today

    - bash: |
                pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
      workingDirectory: $(workingDirectory)
      displayName: 'Install application dependencies'

    - task: ArchiveFiles@2
      displayName: 'Archive files'
      inputs:
        rootFolderOrFile: '$(workingDirectory)'
        includeRootFolder: false
        archiveType: zip
        archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
        replaceExistingArchive: true

    - publish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
      artifact: drop

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build
  condition: succeeded()

  jobs:
  - deployment: Deploy
    displayName: Deploy
    environment: 'development'
    pool:
      vmImage: $(vmImageName)

    strategy:
      runOnce:
        deploy:

          steps:
          - task: AzureFunctionApp@1
            displayName: 'Azure functions app deploy'
            inputs:
              azureSubscription: '$(azureSubscription)'
              appType: functionAppLinux
              appName: $(functionAppName)
              package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'

输出:-

HTTP 触发器已成功部署:-

参考链接:- 我的 SO 线程答案

azure-functions
protobuf==3.20.*

import logging

import azure.functions as func


def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get('name')

    if name:
        return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully.")
    else:
        return func.HttpResponse(
             "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
             status_code=200
        )

trigger:
- master

variables:
  
  azureSubscription: 'xxxxxxxx-xxxxx-xxx9bbd4354dd'

 
  functionAppName: 'valleyfunc541'

 
  vmImageName: 'ubuntu-latest'

 
  workingDirectory: '$(System.DefaultWorkingDirectory)'

stages:
- stage: Build
  displayName: Build stage

  jobs:
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)

    steps:
    - bash: |
        if [ -f extensions.csproj ]
        then
            dotnet build extensions.csproj --runtime ubuntu.16.04-x64 --output ./bin
        fi        
      workingDirectory: $(workingDirectory)
      displayName: 'Build extensions'

    - task: UsePythonVersion@0
      displayName: 'Use Python 3.10'
      inputs:
        versionSpec: 3.10 # Functions V2 supports Python 3.6 as of today

    - bash: |
                pip install --target="./.python_packages/lib/site-packages" -r ./requirements.txt
      workingDirectory: $(workingDirectory)
      displayName: 'Install application dependencies'

    - task: ArchiveFiles@2
      displayName: 'Archive files'
      inputs:
        rootFolderOrFile: '$(workingDirectory)'
        includeRootFolder: false
        archiveType: zip
        archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
        replaceExistingArchive: true

    - publish: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
      artifact: drop

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build
  condition: succeeded()

  jobs:
  - deployment: Deploy
    displayName: Deploy
    environment: 'development'
    pool:
      vmImage: $(vmImageName)

    strategy:
      runOnce:
        deploy:

          steps:
          - task: AzureFunctionApp@1
            displayName: 'Azure functions app deploy'
            inputs:
              azureSubscription: '$(azureSubscription)'
              appType: functionAppLinux
              appName: $(functionAppName)
              package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'

答案2

得分: 0

问题的根本原因，正如 @SiddheshDesai 所指出的，是 Azure Functions 主机加载到缓存中的 protobuf 版本（3.19.6）早于最新的 Tensorflow 所需版本（4.23.3）。

即使将我的 protobuf/TF 导入放在函数应用模块的顶部，仍然出现了相同的错误。

然而，我能够使用 importlib 从我的 site-packages 重新加载 protobuf。

通过将以下代码添加到我的函数来证明了这个原因：

import logging
import importlib
import google.protobuf
logging.info(google.protobuf.__version__)
logging.info(google.protobuf.__file__)
importlib.reload(google.protobuf)
logging.info(google.protobuf.__version__)
logging.info(google.protobuf.__file__)

然而，重新加载会干扰 Azure Functions 库和主机，导致一堆其他异常。

因此，我的结论是这是 Azure Functions 的问题，而不是我的代码问题。

我现在已经将我的函数容器化并重新部署了。这按预期工作。

因此，答案就是，正如 SiddheshDesai 在上面建议的那样，任何需要使用版本大于或等于 3.20.0 的 protobuf 的应用程序都需要容器化，而不是作为代码/从包中运行。

import logging
import importlib
import google.protobuf
logging.info(google.protobuf.__version__)
logging.info(google.protobuf.__file__)
importlib.reload(google.protobuf)
logging.info(google.protobuf.__version__)
logging.info(google.protobuf.__file__)