Tensorflow-Hub 模块未加载

huangapple go评论65阅读模式
英文:

Tensorflow-Hub Modules not loading

问题

我正在尝试使用通用句子编码器 4,它在我家的桌面电脑上可以运行,但我现在不在家,所以必须重新设置一切。我正在一个虚拟环境中使用它。我在代码中没有做任何更改。我目前连接了酒店的WiFi,我认为这可能是一个问题,但我也尝试过使用我的移动热点。通常在我启动程序时它会冻结,就像它正在尝试初始化模块一样,但什么都没有发生,最终会报错。

我已经尝试使用移动热点来解决这个问题,但没有成功。我可以接受任何不涉及从提供的互联网加载的解决方法,它最终会在AWS实例上运行。

这是我正在尝试运行的代码:

import tensorflow_hub as hub
embed = hub.load('https://tfhub.dev/google/universal-sentence-encoder/4')
print('如果这个打印出来,它就加载成功了')

这是错误信息:

l/OneDrive/Desktop/pythontest/Testing environ/test.py"
WARNING:absl:Deleting lock file C:\Users\reill\AppData\Local\Temp\tfhub_modules3d866c06683311b44b4992fd46003be952409c.lock due to inactivity.
Traceback (most recent call last):
  File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\resolver.py", line 192, in download_and_uncompress
    file_utils.extract_tarfile_to_destination(
  File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\file_utils.py", line 52, in extract_tarfile_to_destination       
    extract_file(tgz, tarinfo, abs_target_path, log_function=log_function)
  File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\file_utils.py", line 35, in extract_file
    buf = src.read(buffer_size)
          ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\reill\AppData\Local\Programs\Python\Python311\Lib\tarfile.py", line 696, in readinto
    buf = self.read(len(b))
          ^^^^^^^^^^^^^^^^^
  File "C:\Users\reill\AppData\Local\Programs\Python\Python311\Lib\tarfile.py", line 687, in read
    raise ReadError("unexpected end of data")
tarfile.ReadError: unexpected end of data

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\test.py", line 2, in <module>
    embed =  hub.load(
             ^^^^^^^^^
  File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\module_v2.py", line 93, in load
    module_path = resolve(handle)
                  ^^^^^^^^^^^^^^^
  File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\module_v2.py", line 48, in resolve
    return registry.resolver(handle)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\registry.py", line 49, in __call__
    return impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\compressed_module_resolver.py", line 67, in __call__
    return resolver.atomic_download(handle, download, module_dir,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\resolver.py", line 421, in atomic_download
    download_fn(handle, tmp_dir)
  File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\compressed_module_resolver.py", line 64, in download
    return resolver.DownloadManager(handle).download_and_uncompress(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\resolver.py", line 200, in download_and_uncompress
    raise IOError("%s does not appear to be a valid module." % self._url)
OSError: https://tfhub.dev/google/universal-sentence-encoder/4 does not appear to be a valid module.
英文:

I am trying to use the universal sentence encoder 4, It works on my desktop at home but I am not home so I am having to re-set everything up. I am using this all in a virtual environment. I have changed nothing in the code. I am on hotel wifi I believe that could be an issue but I also tried it on my mobile hotspot. It typically just freezes as I start the program as if it is trying to initialize the module but nothing happens and it eventually spits out an error.

I have tried using a mobile hotspot to get around this problem and that did not work. I am okay with any kind of work around that doesn't envolve loading it from the internet provided, it would work on an aws instance eventually.

Here is the code I am trying to run:

import tensorflow_hub as hub
embed =  hub.load(
            &#39;https://tfhub.dev/google/universal-sentence-encoder/4&#39;)
print(&#39;if this prints it actually loaded&#39;)

Here is the error:
l/OneDrive/Desktop/pythontest/Testing environ/test.py"
WARNING:absl:Deleting lock file C:\Users\reill\AppData\Local\Temp\tfhub_modules\063d866c06683311b44b4992fd46003be952409c.lock due to inactivity.
Traceback (most recent call last):
File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\resolver.py", line 192, in download_and_uncompress
file_utils.extract_tarfile_to_destination(
File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\file_utils.py", line 52, in extract_tarfile_to_destination
extract_file(tgz, tarinfo, abs_target_path, log_function=log_function)
File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\file_utils.py", line 35, in extract_file
buf = src.read(buffer_size)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\reill\AppData\Local\Programs\Python\Python311\Lib\tarfile.py", line 696, in readinto
buf = self.read(len(b))
^^^^^^^^^^^^^^^^^
File "C:\Users\reill\AppData\Local\Programs\Python\Python311\Lib\tarfile.py", line 687, in read
raise ReadError("unexpected end of data")
tarfile.ReadError: unexpected end of data

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\test.py", line 2, in <module>
embed = hub.load(
^^^^^^^^^
File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\module_v2.py", line 93, in load
module_path = resolve(handle)
^^^^^^^^^^^^^^^
File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\module_v2.py", line 48, in resolve
return registry.resolver(handle)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\registry.py", line 49, in call
return impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\compressed_module_resolver.py", line 67, in call
return resolver.atomic_download(handle, download, module_dir,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\resolver.py", line 421, in atomic_download
download_fn(handle, tmp_dir)
File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\compressed_module_resolver.py", line 64, in download
return resolver.DownloadManager(handle).download_and_uncompress(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\reill\OneDrive\Desktop\pythontest\Testing environ\env\Lib\site-packages\tensorflow_hub\resolver.py", line 200, in download_and_uncompress
raise IOError("%s does not appear to be a valid module." % self._url)
OSError: https://tfhub.dev/google/universal-sentence-encoder/4 does not appear to be a valid module.

答案1

得分: 1

我能够在Google Colab中使用Python 3.10下载TF Hub universal-sentence-encoder/4模块,在conda环境中使用Python 3.11也可以。看起来这个OSError可能是由于网络问题引起的,正如你提到的。TF Hub中有一些常见问题,如下所示 -

  1. "协议违规导致EOF" - 当安装的Python版本不支持托管模块的服务器的TLS要求时发生
  2. "无法验证tfhub.dev的证书" - 如果网络上的某些内容试图充当dev gTLD,则可能会生成这个错误。
  3. 无法写入缓存目录 /tmp/tfhub_modules

你可以尝试通过模拟在URL中附加 ?tf-hub-format=compressed 协议来手动下载此模块,以下载一个tar压缩文件,必须手动解压缩到本地文件中。

# 为TF Hub模块创建一个文件夹。
!mkdir /tmp/moduleA

# 下载模块,并将其解压缩到目标文件夹。你可能需要手动执行此操作。
!curl -L "https://tfhub.dev/google/universal-sentence-encoder/4?tf-hub-format=compressed" | tar -zxvC /tmp/moduleA

# 测试以确保它可以正常工作。
import tensorflow_hub as hub
hub.Module("/tmp/moduleA")

或者

import tensorflow_hub as hub
embed =  hub.load('https://tfhub.dev/google/universal-sentence-encoder/4?tf-hub-format=compressed')
print('如果这样打印,它实际上已加载')

请参考这个类似的问题以供参考。

英文:

I am able to download the TF Hub universal-sentence-encoder/4 module in Google Colab using Python 3.10 and in conda environment using Python 3.11.<br>
It seems this OSError could be due to network issue as you mentioned. There are some common issues in TF Hub as below -
<br>1. "EOF occurred in violation of protocol" - when the installed python version does not support the TLS requirements of the server hosting the module
<br>2. "cannot verify tfhub.dev's certificate" - which is likely to be generated if something on the network is trying to act as the dev gTLD.
<br>3. Failures to write to the cache directory /tmp/tfhub_modules

You can try to manually download this module by simulating the protocol of attaching ?tf-hub-format=compressed to the URL to download a tar compressed file that has to be manually decompressed into a local file

# Create a folder for the TF hub module.
!mkdir /tmp/moduleA

#Download the module, and uncompress it to the destination folder. You might want to do this manually.
!curl -L &quot;https://tfhub.dev/google/universal-sentence-encoder/4?tf-hub-format=compressed&quot; | tar -zxvC /tmp/moduleA

#Test to make sure it works.
import tensorflow_hub as hub
hub.Module(&quot;/tmp/moduleA&quot;)

or

import tensorflow_hub as hub
embed =  hub.load(
            &#39;https://tfhub.dev/google/universal-sentence-encoder/4?tf-hub-format=compressed&#39;)
print(&#39;if this prints it actually loaded&#39;)

Please have a look at this similar issue for your reference.

huangapple
  • 本文由 发表于 2023年6月13日 03:00:22
  • 转载请务必保留本文链接:https://go.coder-hub.com/76459557.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定