英文:
Undefined symbol when importing tf-sentencepiece
问题
要解决这个错误并成功启动sentencepiece，您可以尝试以下步骤：
-
检查 TensorFlow 版本兼容性：错误信息中提到了一个未定义的符号，可能是 TensorFlow 和 tf-sentencepiece 版本之间的兼容性问题。请确保您的 TensorFlow 版本与 tf-sentencepiece 兼容。您可以尝试升级 TensorFlow 到与 tf-sentencepiece 兼容的版本。
-
重新构建 Docker 镜像：在您的 Dockerfile 中，重新构建 Docker 镜像，以确保所有依赖项正确安装。使用以下命令重新构建：
docker-compose build -
清除 Pipenv 虚拟环境：有时候，虚拟环境中的依赖项可能导致问题。您可以尝试删除 Pipenv 虚拟环境并重新创建它：
pipenv --rm pipenv install -
查看其他依赖项：检查您的 Django 项目是否还有其他依赖项，可能需要特定版本或配置。确保所有依赖项都正确安装。
-
查看 Docker 容器日志：查看 Docker 容器的日志以获取更多详细信息，以便确定出错的具体位置。可以使用以下命令查看容器日志：
docker-compose logs -
更新 tf-sentencepiece：检查是否有新的 tf-sentencepiece 版本可用，并尝试更新到最新版本，以查看是否解决了问题。
-
与社区寻求帮助：如果上述步骤都没有解决问题，您可以尝试在相关社区或论坛上寻求帮助，例如 TensorFlow 或 tf-sentencepiece 的官方论坛或GitHub存储库。
请按照上述步骤逐一尝试，看看哪个步骤能够解决您的问题。希望这些建议对您有帮助！
英文:
On my MacBook (version 10.14.6) I am succesfully running a Django application including TensorFlow and tf-sentencepiece (in particular to use the universal sentence encoder model). When I perform a pipenv lock -r > requirements.txt I get the following required packages:
-i https://pypi.org/simple
absl-py==0.8.1
aniso8601==7.0.0
astor==0.8.0
bert-embedding==1.0.1
boto3==1.10.34
boto==2.49.0
botocore==1.13.34
certifi==2019.11.28
chardet==3.0.4
coreapi==2.3.3
coreschema==0.0.4
cycler==0.10.0
defusedxml==0.6.0
diff-match-patch==20181111
django-cors-headers==3.2.0
django-elasticsearch-dsl-drf==0.18
django-elasticsearch-dsl==0.5.1
django-filter==2.2.0
django-import-export==2.0
django-neomodel==0.0.4
django-nine==0.2.2
django-pandas==0.6.1
django-rest-framework==0.1.0
django-rest-swagger==2.2.0
django==2.2.8
djangorestframework==3.10.3
docutils==0.15.2
elasticsearch-dsl==6.1.0
elasticsearch==6.4.0
et-xmlfile==1.0.1
gast==0.3.2
gensim==3.8.1
gluonnlp==0.6.0
graphene-neo4j==2.2.3.9
graphene==2.1.8
graphql-core==2.2.1
graphql-relay==2.0.1
graphviz==0.8.4
grpcio==1.25.0
h5py==2.10.0
idna==2.8
ipaddress==1.0.23
itypes==1.1.0
jdcal==1.4.1
jinja2==2.10.3
jmespath==0.9.4
joblib==0.14.0
keras-applications==1.0.8
keras-preprocessing==1.1.0
kiwisolver==1.1.0
lazy-import==0.2.2
lxml==4.4.2
markdown==3.1.1
markuppy==1.14
markupsafe==1.1.1
matplotlib==3.1.2
mock==3.0.5
mxnet==1.4.0
neo4j-driver==1.6.3
neomodel-next==3.5.2.5
neomodel==3.3.1
neotime==1.7.4
nltk==3.4.5
numpy==1.14.6
odfpy==1.4.0
openapi-codec==1.3.2
openpyxl==3.0.2
pandas==0.25.3
promise==2.2.1
protobuf==3.11.1
pycld2==0.41
pyparsing==2.4.5
python-dateutil==2.8.0
pytorch-pretrained-bert==0.6.2
pytz==2019.3
pyyaml==5.2
regex==2019.11.1
requests==2.22.0
risparser==0.4.3
rx==1.6.1
s3transfer==0.2.1
scikit-learn==0.22
scipy==1.3.3
simplejson==3.17.0
singledispatch==3.4.0.3
six==1.13.0
sklearn==0.0
smart-open==1.9.0
sqlalchemy==1.3.11
sqlparse==0.3.0
tablib==0.14.0
tensorboard==1.13.1
tensorflow-estimator==1.13.0
tensorflow-hub==0.7.0
tensorflow==1.13.1
termcolor==1.1.0
tf-sentencepiece==0.1.82
torch==1.3.1
tqdm==4.40.1
typing==3.6.6
uritemplate==3.0.0
urllib3==1.25.7
werkzeug==0.16.0
wheel==0.33.6 ; python_version >= '3'
xlrd==1.2.0
xlwt==1.3.0
The above list is also generated in my Dockerfile:
# The first instruction is what image we want to base our container on
# We Use an official Python runtime as a parent image
FROM python:3.7-stretch
# The enviroment variable ensures that the python output is set straight
# to the terminal with out buffering it first
ENV PYTHONUNBUFFERED 1
# create root directory for our project in the container
RUN mkdir /var/lib/django
# Set the working directory to /var/lib/django
WORKDIR /var/lib/django
# Copy the current directory contents into the container at /var/lib/django
ADD . /var/lib/django/
# Install any needed packages specified in pipfile using piping
RUN pip install --upgrade pip
RUN pip install pipenv
# Below are packages that need to be added to allow C tools
RUN apt update
RUN apt-get -y install libc-dev
RUN apt-get -y install build-essential
# Install requirements
RUN pipenv lock -r > requirements.txt
RUN pip install -r requirements.txt
# Run onDockerStartup file (executes python required functions for startup)
RUN python3 onDockerStartup.py
EXPOSE 8000
CMD ["python3", "manage.py", "runserver", "0.0.0.0:8000", "--noreload"]
However, when I start my application with docker-compose I get the following error:
django_1 | Traceback (most recent call last):
django_1 | File "manage.py", line 21, in <module>
django_1 | main()
django_1 | File "manage.py", line 17, in main
django_1 | execute_from_command_line(sys.argv)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
django_1 | utility.execute()
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
django_1 | self.fetch_command(subcommand).run_from_argv(self.argv)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
django_1 | self.execute(*args, **cmd_options)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 60, in execute
django_1 | super().execute(*args, **options)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
django_1 | output = self.handle(*args, **options)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 95, in handle
django_1 | self.run(**options)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 104, in run
django_1 | self.inner_run(None, **options)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 117, in inner_run
django_1 | self.check(display_num_errors=True)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 390, in check
django_1 | include_deployment_checks=include_deployment_checks,
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 377, in _run_checks
django_1 | return checks.run_checks(**kwargs)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/checks/registry.py", line 72, in run_checks
django_1 | new_errors = check(app_configs=app_configs)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/checks/urls.py", line 40, in check_url_namespaces_unique
django_1 | all_namespaces = _load_all_namespaces(resolver)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/core/checks/urls.py", line 57, in _load_all_namespaces
django_1 | url_patterns = getattr(resolver, 'url_patterns', [])
django_1 | File "/usr/local/lib/python3.7/site-packages/django/utils/functional.py", line 80, in __get__
django_1 | res = instance.__dict__[self.name] = self.func(instance)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/urls/resolvers.py", line 584, in url_patterns
django_1 | patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/utils/functional.py", line 80, in __get__
django_1 | res = instance.__dict__[self.name] = self.func(instance)
django_1 | File "/usr/local/lib/python3.7/site-packages/django/urls/resolvers.py", line 577, in urlconf_module
django_1 | return import_module(self.urlconf_name)
django_1 | File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
django_1 | return _bootstrap._gcd_import(name[level:], package, level)
django_1 | File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
django_1 | File "<frozen importlib._bootstrap>", line 983, in _find_and_load
django_1 | File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
django_1 | File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
django_1 | File "<frozen importlib._bootstrap_external>", line 728, in exec_module
django_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
django_1 | File "/var/lib/django/backend/urls.py", line 29, in <module>
django_1 | path(r'search/', include('apps.search.urls')),
django_1 | File "/usr/local/lib/python3.7/site-packages/django/urls/conf.py", line 34, in include
django_1 | urlconf_module = import_module(urlconf_module)
django_1 | File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
django_1 | return _bootstrap._gcd_import(name[level:], package, level)
django_1 | File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
django_1 | File "<frozen importlib._bootstrap>", line 983, in _find_and_load
django_1 | File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
django_1 | File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
django_1 | File "<frozen importlib._bootstrap_external>", line 728, in exec_module
django_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
django_1 | File "/var/lib/django/apps/search/urls.py", line 4, in <module>
django_1 | from . import views
django_1 | File "/var/lib/django/apps/search/views.py", line 24, in <module>
django_1 | from .servicelayer.question_answer import find_answer_to_question
django_1 | File "/var/lib/django/apps/search/servicelayer/question_answer.py", line 1, in <module>
django_1 | from ml.use import embed_use
django_1 | File "/var/lib/django/ml/use.py", line 9, in <module>
django_1 | import tf_sentencepiece
django_1 | File "/usr/local/lib/python3.7/site-packages/tf_sentencepiece/__init__.py", line 5, in <module>
django_1 | from tf_sentencepiece.sentencepiece_processor_ops import *
django_1 | File "/usr/local/lib/python3.7/site-packages/tf_sentencepiece/sentencepiece_processor_ops.py", line 43, in <module>
django_1 | _gen_sentencepiece_processor_op = tf.load_op_library(so_file)
django_1 | File "/usr/local/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library
django_1 | lib_handle = py_tf.TF_LoadLibrary(library_filename)
django_1 | tensorflow.python.framework.errors_impl.NotFoundError: /usr/local/lib/python3.7/site-packages/tf_sentencepiece/_sentencepiece_processor_ops.so.1.13.1: undefined symbol: _ZN10tensorflow12OpDefBuilder4AttrESs
How should I solve this error to get sentencepiece to start successfully?
答案1
得分: 2
问题不在Dockerfile的设置上，也不在requirements.txt中的结果包上。问题与Tensorflow不兼容Python 3.7有关（尽管奇怪的是一切在本地都正常运行）。我在本地机器上切换到Python 3.6.5来测试一切是否正常工作。然后，我更改了我的Dockerfile以从以下版本开始：
FROM python:3.6-stretch
我的应用程序现在可以成功运行。
英文:
The problem was not in the setup of the Dockerfile, nor in the resulting packages in the requirements.txt. It was related to Tensorflow not being compatible with Python 3.7 (still strange that everything was working fine locally though). I moved to Python 3.6.5 on my local machine to test whether everything was working. Then I changed my Dockerfile to start with
FROM python:3.6-stretch
My application could now run successfully.
答案2
得分: 0
I have no skills in Django, but it seems that tensorflow is trying to find a package (with a strange name) and failing.
我对Django没有任何经验，但似乎tensorflow正在尝试查找一个奇怪名称的包并失败。
I'd first suggest to try and fix your docker container setup, and check that pipenv lock -r yield the same result inside and outside your container.
我首先建议尝试修复您的Docker容器设置，并检查pipenv lock -r在容器内外是否产生相同的结果。
- as you said in the commentaries, on the host pc
正如您在评论中所说，在主机上
# Install requirements
RUN pipenv lock -r > requirements.txt
- Replicate this env in the Docker container
在Docker容器中复制这个环境
# Dockerfile
#...
# Install requirements
ADD requirements.txt .
RUN pip install -r requirements.txt
- to check:
进行检查：
docker container run -it --rm <your image> pipenv lock -r
That is as far as I can take you, I hope it'll be enough.
这就是我能为您提供的帮助，希望足够了。
英文:
I have no skills in Django, but it seems that tensorflow is trying to find a package (with a strange name) and failing.
I'd first suggest to try and fix your docker container setup, and check that pipenv lock -r yield the same result inside and outside your container.
- as you said in the commentaries, on the host pc
# Install requirements
RUN pipenv lock -r > requirements.txt
- Replicate this env in the Docker container
# Dockerfile
#...
# Install requirements
ADD requirements.txt .
RUN pip install -r requirements.txt
- to check:
docker container run -it --rm <your image> pipenv lock -r
That is as far as I can take you, I hope it'll be enough
答案3
得分: 0
Dockerfile
如评论中所述，首先应该在主机上生成您的 requirements.txt 文件，而不是在 Dockerfile 中。
# 使用管道安装指定在 pipfile 中的所需包
RUN pip install --upgrade pip
RUN pip install pipenv
...
# 安装依赖项
RUN pipenv lock -r > requirements.txt
这将粗略生成一个仅包含 pip 和 pipenv 的 requirements.txt。
因此，在主机上生成您的 requirements.txt 并将其添加到容器中：
# 安装依赖项
ADD requirements.txt .
RUN pip install -r requirements.txt
SentencePiece
然后，您的错误似乎与一个已编译的库有关。
Python 正在查找名为 _ZN10tensorflow12OpDefBuilder4AttrESs 的符号，位于 _sentencepiece_processor_ops.so.1.13.1 库中，但未找到。
这通常发生在版本不匹配的情况下 - 您在旧版本的库中寻找新符号。
我不熟悉 sentencepiece，但我看到有两个 Python 包：sentencepiece 和 tf_sentencepiece。
但是，我看到 sentencepiece 没有列在您的 requirements.txt 中（也许您手动安装了它），而且我感到 tf_sentencepiece 需要它才能正常工作。
此外，我发现您有 _sentencepiece_processor_ops.so.1.13.1，而 sentencepiece 的最新版本是 0.1.85。
英文:
Dockerfile
As stated in comments, first you requirements.txt file should be generated on the host machine, not in you Dockerfile.
# Install any needed packages specified in pipfile using piping
RUN pip install --upgrade pip
RUN pip install pipenv
...
# Install requirements
RUN pipenv lock -r > requirements.txt
Will roughly generate a requirements.txt with only pip and pipenv inside.
So, generate your requirements.txt on your host and add it to your container:
# Install requirements
ADD requirements.txt .
RUN pip install -r requirements.txt
SentencePiece
Then, your error seems related to a compiled library.
Python is looking for a symbol named _ZN10tensorflow12OpDefBuilder4AttrESs inside the _sentencepiece_processor_ops.so.1.13.1 library, but it does not found it.
This generally happens when you have some version mismatch -- you are looking for a new symbol in a old version of library.
I don't know sentencepiece but I see that it exist two python packages: sentencepiece and tf_sentencepiece.
However, I see that sentencepiece is not listed in your requirements.txt (maybe you install it manually), and I feel that tf_sentencepiece requires it to work.
Moreover, I find it strange that you have _sentencepiece_processor_ops.so.1.13.1 while sentencepiece last version is 0.1.85.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论