conda: 如果我们将pip_interop_enabled设置为True，会有什么不同？

huangapple
go评论63阅读模式
英文:

conda: what difference does it make if we set pip_interop_enabled=True?

问题

有很多帖子在这个网站上提到，在某些环境中设置pip_interop_enabled=True的想法，通常是顺带提到的。据我所知，人们说这可以让conda和pip3更好地互动。确切地说，如果这是真的，人们说conda将在PyPI中搜索那些在主要渠道中不存在的包。他们还说这是“实验性的”。

以下是conda的文档。文档指出，即使在pip_interop_enabled=False的情况下，最近版本中conda的许多行为也有所改进，这引发了对这个设置究竟是什么意思的疑问。

这里是我的问题：从实际角度来看，所有这些意味着什么？

  • 唯一的区别是如果设置为True，conda会搜索PyPI，否则不会吗？

  • 它还有其他功能吗？例如，如果我需要从pip安装某个包，如果设置为True，conda会更明智地避免覆盖它吗？

  • 具体而言，如果我将其设置为True，会出现什么问题？是否已知存在某些边缘情况，如果设置为True，会导致一些问题？

  • 为什么我会不想设置这个选项？

英文:

There are many posts on this site which reference, typically in passing, the idea of setting pip_interop_enabled=True within some environment. This makes conda and pip3 somehow interact better, I am told. To be precise, people say conda will search PyPI for packages that don't exist in the main channels if this is true. They also say it's "experimental."

Here is conda's documentation about this. It notes that much of conda's behavior in recent versions has also improved even with pip_interop_enabled=False, leading to questions about what this setting even does.

Here is my question: in real terms, what does all of this mean?

  • Is the only difference that conda will search PyPI if this is True and not if it's False?

  • Are there other things that it does? For instance, if I need to install some package from pip, will conda know better not to clobber it if this setting is True?

  • What, to be precise, goes wrong if I set this to True? Are there known edge cases that somehow break things if this "experimental" setting is set to True?

  • Why would I ever not want to set this?

答案1

得分: 5

不是 PyPI 搜索功能

首先，让我们澄清一下：Conda 不会 "搜索 PyPI" - 这不是 pip_interop_enabled 配置选项添加的功能。相反，它使 solver 能够允许已经使用 pip 安装的软件包来满足 Conda 软件包的依赖要求。请注意，该选项关于 Pip 互操作性（与 PyPI 不同）并且不管软件包是否来自 PyPI、GitHub、本地等等都没有关系。

示例：scipy -> numpy

让我们考虑一个简单的示例来说明这个行为。首先，我们有一个具有 Python 3.10 和从 PyPI 安装的 numpy 的环境。

pip_interop.yaml

name: pip_interop
channels:
  - conda-forge
dependencies:
  - python=3.10
  - pip

  ## PyPI packages
  - pip:
    - numpy

我们可以使用以下命令创建该环境：

conda env create -n pip_interop -f pip_interop.yaml

并验证 numpy 来自 PyPI：

$ conda list -n pip_interop numpy
# packages in environment at /Users/user/mambaforge/envs/pip_interop:
#
# Name                    Version                   Build  Channel
numpy                     1.24.2                   pypi_0

接下来，让我们看看安装 scipy 以及它是如何满足它的 numpy 依赖的。

未启用 Pip 互操作性的安装

在默认模式下，我们会看到以下行为：

$ conda install -n pip_interop scipy
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/user/mambaforge/envs/pip_interop

  added / updated specs:
    - scipy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cryptography-39.0.1        |  py310hdd0c95c_0         1.1 MB
    numpy-1.24.2               |  py310h788a5b3_0         6.1 MB
    scipy-1.10.0               |  py310h240c617_2        20.2 MB
    ------------------------------------------------------------
                                           Total:        27.4 MB

The following NEW packages will be INSTALLED:
...

请注意，尽管环境中已经安装了 numpy，但 Conda 提议用 Conda 版本替换它。也就是说，Conda 只会考虑 conda-meta/ 中的信息来确定软件包是否已安装，而不会检查环境中的 lib/python3.10/site-packages/。

启用 Pip 互操作性的安装

现在，让我们尝试启用 pip_interop_enabled：

$ CONDA_PIP_INTEROP_ENABLED=1 conda install -n foo scipy
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/user/mambaforge/envs/pip_interop

  added / updated specs:
    - scipy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cryptography-39.0.1        |  py310hdd0c95c_0         1.1 MB
    scipy-1.10.0               |  py310h240c617_2        20.2 MB
    ------------------------------------------------------------
                                           Total:        21.3 MB

The following NEW packages will be INSTALLED:
...

现在，numpy 不再被建议替换，这是因为现有的 pip 安装版本被视为能够满足依赖关系。

为什么这是实验性的？

在几年后，这仍然保持实验性的原因可能有多个。一个重要的原因是 Conda 仅测试其软件包构建与依赖项的 Conda 构建之间的兼容性。因此，它不能保证这些软件包在功能上是可以互换的。

此外，Conda 软件包经常引入非 Python 依赖项。有了 wheel 部署的崛起，这是 PyPI 的方法，但并不是普遍适用的。仍然有许多 "包装" 软件包，其中 PyPI 版本假定某个二进制文件在 PATH 上，而 Conda 软件包的安装 保证 该二进制文件也已安装。

另一个重要问题是 PyPI-Conda 名称映射没有明确定义。也就是说，PyPI 中软件包的名称可能与其 Conda 软件包的名称不相符。当名称不匹配时，这可能直接导致在名称不一致时 Conda 不会正确地识别 pip 安装的软件包是否满足要求。因此，互操作性应用方式存在一些意外的异质性。

示例：torch vs pytorch

在 Python 生态系统中，torch 模块由 PyPI 软件包 torch 提供。然而，PyPI 中的 torch 软件包在 Conda 渠道上称为 pytorch。

这就是为什么这可能导致不一致行为的原因。让我们从 PyPI 安装的 torch 开始：

pip_interop.yaml

name: pip_interop
channels:
  - conda-forge
dependencies:
  - python=3.10
  - pip

  ## PyPI packages
  - pip:
    - torch

使用以下命令创建：

conda env create -n pip_interop -f pip_interop.yaml

现在，如果我们即使启用了 pip_interop_enabled，尝试从 Conda 安装 torchvision，我们会得到

英文:

Not a PyPI Searching Feature

First, let's clarify: Conda will not "search PyPI" - that is not what the pip_interop_enabled configuration option adds. Rather, it enables the solver to allow a package already installed with pip to satisfy a dependency requirement of a Conda package. Note that the option is about Pip interoperability (as distinct from PyPI) and it doesn't matter whether the package was sourced from PyPI, GitHub, local, etc..

Example: scipy -> numpy

Let's consider a simple example to illustrate the behavior. Start with the following environment that has Python 3.10 and numpy installed from PyPI.

pip_interop.yaml

name: pip_interop
channels:
  - conda-forge
dependencies:
  - python=3.10
  - pip

  ## PyPI packages
  - pip:
    - numpy

which we can create with

conda env create -n pip_interop -f pip_interop.yaml

and verify that the numpy is from PyPI:

$ conda list -n pip_interop numpy
# packages in environment at /Users/user/mambaforge/envs/pip_interop:
#
# Name                    Version                   Build  Channel
numpy                     1.24.2                   pypi_0

Let's see what would happen installing scipy and in particular, how it satisfies its numpy dependency.

Installing without Pip interoperability

In default mode, we see the following behavior

$ conda install -n pip_interop scipy
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/user/mambaforge/envs/pip_interop

  added / updated specs:
    - scipy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cryptography-39.0.1        |  py310hdd0c95c_0         1.1 MB
    numpy-1.24.2               |  py310h788a5b3_0         6.1 MB
    scipy-1.10.0               |  py310h240c617_2        20.2 MB
    ------------------------------------------------------------
                                           Total:        27.4 MB

The following NEW packages will be INSTALLED:

  appdirs            conda-forge/noarch::appdirs-1.4.4-pyh9f0ad1d_0 
  brotlipy           conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005 
  certifi            conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 
  cffi               conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3 
  charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 
  cryptography       conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0 
  idna               conda-forge/noarch::idna-3.4-pyhd8ed1ab_0 
  libblas            conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas 
  libcblas           conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas 
  libcxx             conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0 
  libgfortran        conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27 
  libgfortran5       conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27 
  liblapack          conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas 
  libopenblas        conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3 
  llvm-openmp        conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0 
  numpy              conda-forge/osx-64::numpy-1.24.2-py310h788a5b3_0 
  packaging          conda-forge/noarch::packaging-23.0-pyhd8ed1ab_0 
  pooch              conda-forge/noarch::pooch-1.6.0-pyhd8ed1ab_0 
  pycparser          conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0 
  pyopenssl          conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0 
  pysocks            conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6 
  python_abi         conda-forge/osx-64::python_abi-3.10-3_cp310 
  requests           conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0 
  scipy              conda-forge/osx-64::scipy-1.10.0-py310h240c617_2 
  urllib3            conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0 

Proceed ([y]/n)?

Observe that despite numpy already being installed in the environment, Conda is proposing to replace it with a Conda version. That is, only considers the information in conda-meta/ to determine whether a package is installed and won't check the environment's lib/python3.10/site-packages/.

Installing with Pip interoperability

Now we try it with the pip_interop_enabled turned on:

$ CONDA_PIP_INTEROP_ENABLED=1 conda install -n foo scipy
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/user/mambaforge/envs/pip_interop

  added / updated specs:
    - scipy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cryptography-39.0.1        |  py310hdd0c95c_0         1.1 MB
    scipy-1.10.0               |  py310h240c617_2        20.2 MB
    ------------------------------------------------------------
                                           Total:        21.3 MB

The following NEW packages will be INSTALLED:

  appdirs            conda-forge/noarch::appdirs-1.4.4-pyh9f0ad1d_0 
  brotlipy           conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005 
  certifi            conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 
  cffi               conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3 
  charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 
  cryptography       conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0 
  idna               conda-forge/noarch::idna-3.4-pyhd8ed1ab_0 
  libblas            conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas 
  libcblas           conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas 
  libcxx             conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0 
  libgfortran        conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27 
  libgfortran5       conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27 
  liblapack          conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas 
  libopenblas        conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3 
  llvm-openmp        conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0 
  packaging          conda-forge/noarch::packaging-23.0-pyhd8ed1ab_0 
  pooch              conda-forge/noarch::pooch-1.6.0-pyhd8ed1ab_0 
  pycparser          conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0 
  pyopenssl          conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0 
  pysocks            conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6 
  python_abi         conda-forge/osx-64::python_abi-3.10-3_cp310 
  requests           conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0 
  scipy              conda-forge/osx-64::scipy-1.10.0-py310h240c617_2 
  urllib3            conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0 


Proceed ([y]/n)?

Note that now the numpy is not proposed to be replaced and this is because the existing pip-installed version is consider able to satisfy the dependency.

Why is this experimental?

There may be multiple reasons why this remains experimental after several years. One important reason is that Conda only tests its package builds against Conda builds of the dependencies. So, it cannot guarantee that the packages are functionally exchangeable.

Furthermore, Conda packages often bring in non-Python dependencies. There has been a rise in wheel deployments, which is the PyPI approach to this, but isn't ubiquitous. There are still many "wrapper" packages out there where the PyPI version assumes some binary is on PATH, whereas the installation of the Conda package guarantees the binary is also installed.

Another important issue is that the PyPI-Conda name mapping is not well-defined. That is, the name of a package in PyPI may not correspond to its Conda package name. This can directly lead to cryptic issues when the names diverge. Specifically, Conda will not correctly recognize that a pip-installed package satisfies the requirement when the names don't match. Hence, the is some unexpected heterogeneity in how the interoperability applies.

Example: torch vs pytorch

In the Python ecosystem, the torch module is provided by the PyPI package torch. However, the package torch in PyPI goes by pytorch on Conda channels.

Here's how this can lead to inconsistent behavior. Let's begin with torch installed from PyPI:

pip_interop.yaml

name: pip_interop
channels:
  - conda-forge
dependencies:
  - python=3.10
  - pip

  ## PyPI packages
  - pip:
    - torch

Creating with:

conda env create -n pip_interop -f pip_interop.yaml

Now if we install torchvision from Conda, even with the pip_interop_enabled on, we get:

$ CONDA_PIP_INTEROP_ENABLED=1 conda install -n pip_interop torchvision
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/user/mambaforge/envs/pip_interop

  added / updated specs:
    - torchvision


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cryptography-39.0.1        |  py310hdd0c95c_0         1.1 MB
    jpeg-9e                    |       hb7f2c08_3         226 KB
    libprotobuf-3.21.12        |       hbc0c0cd_0         1.8 MB
    mkl-2022.2.1               |   h44ed08c_16952       113.1 MB
    numpy-1.24.2               |  py310h788a5b3_0         6.1 MB
    pillow-9.4.0               |  py310h306a057_1        44.1 MB
    pytorch-1.13.1             |cpu_py310h2bbf33f_1        56.9 MB
    sleef-3.5.1                |       h6db0672_2         1.0 MB
    torchvision-0.14.1         |cpu_py310hd5ee960_0         5.9 MB
    ------------------------------------------------------------
                                           Total:       230.1 MB

The following NEW packages will be INSTALLED:

  brotlipy           conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005 
  certifi            conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 
  cffi               conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3 
  charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 
  cryptography       conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0 
  freetype           conda-forge/osx-64::freetype-2.12.1-h3f81eb7_1 
  idna               conda-forge/noarch::idna-3.4-pyhd8ed1ab_0 
  jpeg               conda-forge/osx-64::jpeg-9e-hb7f2c08_3 
  lcms2              conda-forge/osx-64::lcms2-2.14-h29502cd_1 
  lerc               conda-forge/osx-64::lerc-4.0.0-hb486fe8_0 
  libblas            conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas 
  libcblas           conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas 
  libcxx             conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0 
  libdeflate         conda-forge/osx-64::libdeflate-1.17-hac1461d_0 
  libgfortran        conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27 
  libgfortran5       conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27 
  liblapack          conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas 
  libopenblas        conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3 
  libpng             conda-forge/osx-64::libpng-1.6.39-ha978bb4_0 
  libprotobuf        conda-forge/osx-64::libprotobuf-3.21.12-hbc0c0cd_0 
  libtiff            conda-forge/osx-64::libtiff-4.5.0-hee9004a_2 
  libwebp-base       conda-forge/osx-64::libwebp-base-1.2.4-h775f41a_0 
  libxcb             conda-forge/osx-64::libxcb-1.13-h0d85af4_1004 
  llvm-openmp        conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0 
  mkl                conda-forge/osx-64::mkl-2022.2.1-h44ed08c_16952 
  numpy              conda-forge/osx-64::numpy-1.24.2-py310h788a5b3_0 
  openjpeg           conda-forge/osx-64::openjpeg-2.5.0-h13ac156_2 
  pillow             conda-forge/osx-64::pillow-9.4.0-py310h306a057_1 
  pthread-stubs      conda-forge/osx-64::pthread-stubs-0.4-hc929b4f_1001 
  pycparser          conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0 
  pyopenssl          conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0 
  pysocks            conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6 
  python_abi         conda-forge/osx-64::python_abi-3.10-3_cp310 
  pytorch            conda-forge/osx-64::pytorch-1.13.1-cpu_py310h2bbf33f_1 
  requests           conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0 
  sleef              conda-forge/osx-64::sleef-3.5.1-h6db0672_2 
  tbb                conda-forge/osx-64::tbb-2021.7.0-hb8565cd_1 
  torchvision        conda-forge/osx-64::torchvision-0.14.1-cpu_py310hd5ee960_0 
  typing_extensions  conda-forge/noarch::typing_extensions-4.4.0-pyha770c72_0 
  urllib3            conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0 
  xorg-libxau        conda-forge/osx-64::xorg-libxau-1.0.9-h35c211d_0 
  xorg-libxdmcp      conda-forge/osx-64::xorg-libxdmcp-1.1.3-h35c211d_0 
  zstd               conda-forge/osx-64::zstd-1.5.2-hbc0c0cd_6 


Proceed ([y]/n)?

That is, Conda still tries to install pytorch and this means that it will lead to clobbering of the existing torch package installed from PyPI. This has the potential to having residual files from the clobbered version of the package intermixed with the clobbering version.

Basically, this is undefined behavior and the Conda software may not give you any warning about potential problems.

huangapple
  • 本文由 发表于 2023年2月16日 11:00:38
  • 转载请务必保留本文链接：https://go.coder-hub.com/75467411.html
  • anaconda
  • conda
  • pip
  • pypi
  • python
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定