Python Wheel 包括共享库,构建为纯Python、跨平台、非特定平台

huangapple go评论67阅读模式
英文:

Python Wheel that includes shared library is built as pure-Python platform independent none-any

问题

因为CUDA的存在,您似乎无法轻松使用传统的setuptools扩展方式来构建Wheel包,所以您选择了预编译代码到共享库,并希望将它们包含在Wheel包中。但是,当您构建Wheel包时,它包含了共享库,但仍然将输出的Wheel包标记为"none-any"扩展,表明它是纯Python和平台无关的,这是不正确的。因此,cibuildwheel拒绝对其运行auditwheel。您已经阅读了一些关于强制将Wheel标记为平台相关的帖子,但您认为您不应该强制这样做,可能是您做错了一些事情。

目录结构:

  • mypackage
    • pyproject.toml
    • setup.cfg
    • src
      • mypackage
        • __init__.py
        • aaa.py
        • bbb.c
        • ccc.cu
        • libmypackage_cpu.so
        • libmypackage_cuda.so
        • before_all.sh(用于构建.so文件的脚本)

pyproject.toml:

[build-system]
requires = ["setuptools>=42"]
build-backend = "setuptools.build_meta"

[tool.cibuildwheel]
build = "cp36-*"
skip = ["pp*", "*686"]
manylinux-x86_64-image = "manylinux2014"

[tool.cibuildwheel.linux]
before-all = "bash {project}/src/mypackage/before_all.sh"

setup.cfg:

[metadata]
name = mypackage

[options]
package_dir =
    = src
include_package_data = True
zip_safe = False
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src

[options.package_data]
mypackage =
    *.so

您是否遗漏了一些内容,导致打包工具无法检测到Wheel包不是纯Python且平台相关,或者强制将其标记为平台相关(例如,创建一个空的Extension)是否是正常的做法?

英文:

I wish to use some C and CUDA code in my Python package (which I then call using ctypes). Because of the CUDA, it doesn't seem to be easy to use the traditional approach of a setuptools Extension, so I instead pre-compile the code to shared libraries and then wish to include them in a Wheel. When I build the Wheel it includes the shared libraries, but it still gives the output Wheel a "none-any" extension, indicating that it is pure Python and platform independent, which is not correct. cibuildwheel then refuses to run auditwheel on it because of this. I have read posts about forcing Wheels to be labelled as platform dependent, but I imagine that I shouldn't have to force it and that I am instead doing something wrong.

Directory structure:

  • mypackage
    • pyproject.toml
    • setup.cfg
    • src
      • mypackage
        • __init__.py
        • aaa.py
        • bbb.c
        • ccc.cu
        • libmypackage_cpu.so
        • libmypackage_cuda.so
        • before_all.sh (script to build the .so files)

pyproject.toml:

[build-system]
requires = ["setuptools>=42"]
build-backend = "setuptools.build_meta"

[tool.cibuildwheel]
build = "cp36-*"
skip = ["pp*", "*686"]
manylinux-x86_64-image = "manylinux2014"

[tool.cibuildwheel.linux]
before-all = "bash {project}/src/mypackage/before_all.sh"

setup.cfg:

[metadata]
name = mypackage

[options]
package_dir =
    = src
include_package_data = True
zip_safe = False
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src

[options.package_data]
mypackage =
    *.so

Am I missing something that is causing the packager to not detect that the Wheel is not pure Python and platform independent, or is forcing the packager to label it otherwise (such as by creating an empty Extension) the normal approach?

答案1

得分: 1

我能够使用@phd提到的方法使其正常工作,并且还能够使用Hatchling而不是Setuptools作为后端来使其正常工作。在这两种情况下,auditwheel都能够成功,找到共享库并包含它们的依赖项。

然而,这两种方法都感觉很脆弱。我意识到Wheels是用于Python扩展的。我不需要我的编译代码对Python有任何了解,虽然我可以使用CFFI或Cython之类的工具来调用我的编译代码,但在我的情况下我看不出任何优势,而且这会引入对Python ABI的依赖。

因此,我决定只提供源代码分发(没有Wheels),在源代码分发中包含所有平台的共享库,并在运行时加载适合平台的正确库。

我应该注意到,有一些Setuptools的扩展也可能满足我的需求,例如setuptools-cuda-cpp用于启用Setuptools编译CUDA,以及setuptools_dso用于构建非Python共享库以在Setuptools中包含Wheels。

如果对其他人有用的话,要让Setuptools和Hatchling生成声明仅需要Python 3(而不是特定版本的CPython)、没有Python ABI依赖性,并且适用于Linux x86_64平台的Wheels,我必须进行以下更改。

Setuptools

setup.py:

from setuptools import setup, Distribution

try:
    from wheel.bdist_wheel import bdist_wheel as _bdist_wheel
    class MyWheel(_bdist_wheel):

        def finalize_options(self):
            _bdist_wheel.finalize_options(self)
            self.root_is_pure = False

        def get_tag(self):
            python, abi, plat = _bdist_wheel.get_tag(self)
            python, abi = 'py3', 'none'
            return python, abi, plat

    class MyDistribution(Distribution):

        def __init__(self, *attrs):
            Distribution.__init__(self, *attrs)
            self.cmdclass['bdist_wheel'] = MyWheel

        def is_pure(self):
            return False

        def has_ext_modules(self):
            return True

except ImportError:
    class MyDistribution(Distribution):
        def is_pure(self):
            return False

        def has_ext_modules(self):
            return True

setup(
    distclass=MyDistribution
)

Hatchling:

hatch_build.py:

from hatchling.builders.hooks.plugin.interface import BuildHookInterface
class CustomHook(BuildHookInterface):
    def initialize(self, version, build_data):
        build_data['pure_python'] = False
        build_data['tag'] = 'py3-none-linux_x86_64'

此外,除了需要对pyproject.toml进行其他更改以使用Hatchling外,还需要包含以下行:

[tool.hatch.build.targets.wheel.hooks.custom]

以运行上述hatch_build.py中的钩子。

英文:

I was able to get it to work using the method referred to by @phd, and I also got it to work using Hatchling instead of Setuptools as the backend. In both cases auditwheel is successful, finding the shared libraries and including their dependencies.

Both methods feel fragile, however. I have come to the realisation that Wheels are intended for Python extensions. I have no need for my compiled code to have any knowledge of Python, and while I could wrap calls to my compiled code using the likes of CFFI or Cython, I don't see an advantage of it in my case and it would introduce a dependence on the Python ABI.

I have thus decided to only provide a source distribution (no Wheels), including the shared libraries for all platforms in the source distribution and loading the correct one for the platform at runtime.

I should note that there are extensions of Setuptools that might also meet my needs, such as setuptools-cuda-cpp to enable Setuptools to compile CUDA, and setuptools_dso to build non-Python shared libraries for inclusion in Wheels with Setuptools.

In case it is useful to someone else, to get Setuptools and Hatchling to produce Wheels that declare themselves to only require Python 3 (not a particular release of CPython), have no Python ABI dependence, and be for the Linux x86_64 platform, I had to make the below changes.

Setuptools:

setup.py:

from setuptools import setup, Distribution

try:
    from wheel.bdist_wheel import bdist_wheel as _bdist_wheel
    class MyWheel(_bdist_wheel):

        def finalize_options(self):
            _bdist_wheel.finalize_options(self)
            self.root_is_pure = False

        def get_tag(self):
            python, abi, plat = _bdist_wheel.get_tag(self)
            python, abi = 'py3', 'none'
            return python, abi, plat

    class MyDistribution(Distribution):

        def __init__(self, *attrs):
            Distribution.__init__(self, *attrs)
            self.cmdclass['bdist_wheel'] = MyWheel

        def is_pure(self):
            return False

        def has_ext_modules(self):
            return True

except ImportError:
    class MyDistribution(Distribution):
        def is_pure(self):
            return False

        def has_ext_modules(self):
            return True

setup(
    distclass=MyDistribution
)

Hatchling:

hatch_build.py:

from hatchling.builders.hooks.plugin.interface import BuildHookInterface
class CustomHook(BuildHookInterface):
    def initialize(self, version, build_data):
        build_data['pure_python'] = False
        build_data['tag'] = 'py3-none-linux_x86_64'

and then, in addition to the other changes to pyproject.toml needed to use Hatchling, the line

[tool.hatch.build.targets.wheel.hooks.custom]

needs to be included to run the hook in the above hatch_build.py.

huangapple
  • 本文由 发表于 2023年6月11日 21:02:37
  • 转载请务必保留本文链接:https://go.coder-hub.com/76450587.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定