英文:
How would you properly use the Python/C API in a C shared library?
问题
Apologies for the lengthy question, but I've translated the content as you requested:
事先为这个非常长的问题道歉 - 这已经是一段旅程了... 所以提前感谢您的耐心。
TL;DR:
当使用以下设置时，Python 会抛出 segfault：
[numpy 代码] <-- [Python/C API] <-- [用 C 实现的 .so 库] <-- [使用 ctypes 的 Python 驱动程序 cdll]
请注意，Python 驱动程序仅用于测试 .so 库，该库最终打算在另一个第三方应用程序中使用。
背景
我们的团队有一些 Python 代码，我们正在尝试将其集成到另一个第三方应用程序中，该应用程序期望以共享对象文件（.so 库）的形式实现符号。为了实现这一目标，我正在尝试创建一个使用 [Python/C API][1] 的 C 共享对象库。我一直在努力使事情按预期运行，所以我试图将问题精简到一个“尽可能简单”的示例：
我正在创建一个基本的 C 共享对象库，使用 Python/C API，特别是此库的方法具有 numpy 依赖性（稍后会重要）。
该库最终打算由一个“运行时应用程序”（通过动态加载符号，例如使用 [dlfcn.h][2]）使用，该应用程序是“预先存在的”（我无法控制编译/链接等）。我遇到了似乎是链接问题的问题，但可能是与 .so 编译方式和/或我如何使用/配置 Python 通过 C API 有关的问题。
我知道这是可能的，因为我在互联网上找到了一些现有用例，但似乎无法确定我的设置在哪里失败。
设置
假设我在 Ubuntu 20.04 上，全局安装了 python3.8-dev（在这个示例中，我实际上是在 Docker 容器中工作，所以如果最终可能是系统设置问题，我很愿意提供 Dockerfile）。此外，我已经在例如 ~/venv 中设置了虚拟环境，它已经被激活，numpy 已安装 - 我可以通过以下方式确认这一点：
(venv) user@4189d31a5bbe:~$ which python && python -c "import numpy; print(numpy)"
/home/user/venv/bin/python
<module 'numpy' from '/home/user/venv/lib/python3.8/site-packages/numpy/__init__.py'>
库文件
我有以下头文件/源文件：
mylibwithpy.h:
#ifndef __MYLIBWITHPY__
#define __MYLIBWITHPY__
#include <stdio.h>
#include <Python.h>
void someFunctionWithPython();
#endif
mylibwithpy.c:
someFunctionWithPython 函数所做的一切就是检查是否初始化了 Python，如果没有初始化，就初始化它，然后尝试导入 numpy。
#include "mylibwithpy.h"
void someFunctionWithPython()
{
if (!Py_IsInitialized())
{
printf("Initializing python...\n");
Py_Initialize();
}
else
{
printf("Python already initialized.\n");
}
printf("Importing numpy...\n");
PyObject* numpy = PyImport_ImportModule("numpy");
if (numpy == NULL)
{
printf("Warning: error during import:\n");
PyErr_Print();
Py_Finalize();
exit(1);
}
return;
}
库的 *.so 文件通过以下 Makefile 目标进行编译：
mylibwithpy.o:
gcc -L/usr/lib/x86_64-linux-gnu -I/usr/include/python3.8 -Wall -c mylibwithpy.c -o $@ -lpython3.8
mylibwithpy.so: mylibwithpy.o
gcc -L/usr/lib/x86_64-linux-gnu -Wall -fPIC -shared -Wl,-soname,$@ -o $@ mylibwithpy.o -lpython3.8
此时，ldd 看起来似乎正常：
(venv) user@4189d31a5bbe:~$ ldd mylibwithpy.so
linux-vdso.so.1 (0x00007ffc36ba6000)
libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f295c5ea000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f295c3f8000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f295c3ca000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f295c3ae000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f295c38b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f295c385000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f295c37e000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f295c22f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f295cb4f000)
示例运行
这是事情开始出错的地方...
首先，让我们尝试一个基本的 Python 驱动程序，使用 Python 的 ctypes 库。
driver.py:
from ctypes import cdll
if __name__ == "__main__":
print("Opening mylibwithpy.so...")
my_so = cdll.LoadLibrary("mylibwithpy.so")
print(".so object: ", my_so)
print(".so object's 'someFunctionWithPython': ", my_so.someFunctionWithPython)
<details>
<summary>英文:</summary>
Apologies in advance for the extremely long question--it has been a journey... so thanks in advance for your patience.
## TL;DR:
python throws a segfault when using this set up:
[numpy code] <-- [python/C api] <-- [C-implemented .so library] <-- [py driver using ctypes' `cdll`]
Note the py driver is just for testing the .so library, the lib is ultimately intended to be used in another 3rd party app.
## Background
Our team has some python code that we are trying to integrate into another 3rd party application, and the application in question expects symbol implementations in the form of a shared object file (.so library). To accomplish this, I am trying to make a shared object library in C that leverages the [Python/C API][1]. I have been running into some issues getting things to run as expected, so I've tried to distill the problem down to an "as-simple-as-possible" example:
I am making a basic shared object library written in C, using the Python/C API, and in particular this library's methods have a `numpy` dependency (this will be important later).
The lib is ultimately intended to be used by a "runtime application" (via dynamically loading symbols, for example with [dlfcn.h][2]) that is "pre-existing" (I don't have control over the compilation/linking etc. of said program).
I am running in to what seems like a linking issue, but may be an issue with either how the so is compiled and/or how I am using/configuring python via the C API.
I know this is possible to do because of some existing use cases I've found about on the internets, but I cannot seem to zero in on where my setup is failing.
## Setup
Let's say I am on ubuntu 20.04 with python3.8-dev installed system-wide (in this example I am actually working in a docker container, so if it ends up possibly a system setup issue I am happy to provide the dockerfile as well).
Further, I have a virtual environment set up in e.g. `~/venv`, it is activated, and numpy is installed--I can confirm this with, for example,
(venv) user@4189d31a5bbe:~$ which python && python -c "import numpy; print(numpy)"
/home/user/venv/bin/python
<module 'numpy' from '/home/user/venv/lib/python3.8/site-packages/numpy/__init__.py'>
### Library Files
I have the following header/source files:
#### `mylibwithpy.h`:
#ifndef __MYLIBWITHPY__
#define __MYLIBWITHPY__
#include <stdio.h>
#include <Python.h>
void someFunctionWithPython();
#endif
#### `mylibwithpy.c`:
All the function `someFunctionWithPython` does is check if python is initialized, initializes if it isn't, and then tries to import `numpy`.
#include "mylibwithpy.h"
void someFunctionWithPython()
{
if (!Py_IsInitialized())
{
printf("Initializing python...\n");
Py_Initialize();
}
else
{
printf("python alread initialized.\n");
}
printf("importing numpy...\n");
PyObject* numpy = PyImport_ImportModule("numpy");
if (numpy == NULL)
{
printf("Warning: error during import:\n");
PyErr_Print();
Py_Finalize();
exit(1);
}
return;
}
The library *.so file is compiled via these **`Makefile`** targets:
mylibwithpy.o:
gcc -L/usr/lib/x86_64-linux-gnu -I/usr/include/python3.8 -Wall -c mylibwithpy.c -o $@ -lpython3.8
mylibwithpy.so: mylibwithpy.o
gcc -L/usr/lib/x86_64-linux-gnu -Wall -fPIC -shared -Wl,-soname,$@ -o $@ mylibwithpy.o -lpython3.8
At this point, `ldd` seems to check out as so far "ok":
(venv) user@4189d31a5bbe:~$ ldd mylibwithpy.so
linux-vdso.so.1 (0x00007ffc36ba6000)
libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f295c5ea000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f295c3f8000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f295c3ca000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f295c3ae000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f295c38b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f295c385000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f295c37e000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f295c22f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f295cb4f000)
## Example runtime
This is where things start going wrong...
First, let's try just a basic python driver, using python's `ctypes` library.
#### `driver.py`
from ctypes import cdll
if __name__ == "__main__":
print("opening mylibwithpy.so...");
my_so = cdll.LoadLibrary("mylibwithpy.so")
print(".so object: ", my_so)
print(".so object's 'someFunctionWithPython': ", my_so.someFunctionWithPython)
print("calling someFunctionWithPython...");
my_so.someFunctionWithPython()
This basic script will result in a `(Segmentation fault)` error at the point that numpy is trying to be imported in the lib function:
(venv) user@4189d31a5bbe:~$ LD_LIBRARY_PATH=. python driver.py
opening mylibwithpy.so...
.so object: <CDLL 'mylibwithpy.so', handle 19f43b0 at 0x7fd873b72610>
.so object's 'someFunctionWithPython': <_FuncPtr object at 0x7fd873ac91c0>
calling someFunctionWithPython...
python alread initialized.
importing numpy...
Segmentation fault (core dumped)
Ok, so I'm not really sure how to even *start* to debug this guy, so let's try again with an equivalent driver in C:
#### `driver.c`:
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>
int main()
{
printf("opening mylibwithpy.so...\n");
void* mylibwithpy_so = dlopen("mylibwithpy.so", RTLD_LAZY);
if (mylibwithpy_so == NULL){
printf("an error occurred during loading mylibwithpy.so: \n%s\n", dlerror());
exit(1);
}
void (*soFunc)();
soFunc = dlsym(mylibwithpy_so, "someFunctionWithPython");
if (soFunc == NULL){
printf("an error occurred during loading symbol someFunctionWithPython: \n%s\n", dlerror());
exit(1);
}
soFunc();
return 0;
}
Compiling this program via:
gcc -L/usr/lib/x86_64-linux-gnu -Wall driver.c -o cdriver -ldl
And running this driver results in an interestingly-much-more verbose error being reported:
(venv) user@4189d31a5bbe:~$ LD_LIBRARY_PATH=. ./cdriver
opening mylibwithpy.so...
Initializing python...
importing numpy...
Warning: error during import:
Traceback (most recent call last):
File "/home/user/venv/lib/python3.8/site-packages/numpy/core/__init__.py", line 23, in <module>
from . import multiarray
File "/home/user/venv/lib/python3.8/site-packages/numpy/core/multiarray.py", line 10, in <module>
from . import overrides
File "/home/user/venv/lib/python3.8/site-packages/numpy/core/overrides.py", line 6, in <module>
from numpy.core._multiarray_umath import (
ImportError: /home/user/venv/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/venv/lib/python3.8/site-packages/numpy/__init__.py", line 141, in <module>
from . import core
File "/home/user/venv/lib/python3.8/site-packages/numpy/core/__init__.py", line 49, in <module>
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.8 from "/home/user/venv/bin/python3"
* The NumPy version is: "1.24.2"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: /home/user/venv/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter
### **AHA!(...?)**
According to this, it seems numpy has some shared objects of its own, but is somehow missing some symbols (PyObject_SelfIter, to be precise--side note, this symbol is listed in the [Python/C API "stable ABI contents"][3]):
/home/user/venv/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter
(Another side note: the listed numpy docs reference https://numpy.org/devdocs/user/troubleshooting-importerror.html does not seem very applicable to the situation or the error I'm getting, but a more discerning eye may find something helpful that I overlooked...)
Also, a quick `ldd` check shows that indeed `libpython` is not among the dynamically linked libraries:
(venv) user@4189d31a5bbe:~$ ldd /home/user/venv/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffed7df2000)
libopenblas64_p-r0-15028c96.3.21.so => /home/user/venv/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-15028c96.3.21.so (0x00007f98a1ae6000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f98a198f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f98a196c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f98a177a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f98a3f29000)
libgfortran-040039e1.so.5.0.0 => /home/user/venv/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libgfortran-040039e1.so.5.0.0 (0x00007f98a12ed000)
libquadmath-96973f99.so.0.0.0 => /home/user/venv/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libquadmath-96973f99.so.0.0.0 (0x00007f98a10ae000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f98a1092000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f98a1077000)
## Testing the linker issue hypothesis
Since we're getting an undefined symbol error printed out in the `cdriver` program, I can try to force-link libpython to the cdriver via:
gcc -L/usr/lib/x86_64-linux-gnu -Wall driver.c -o cdriver -ldl -Wl,--no-as-needed -lpython3.8
and lo and behold this time the program completes without error:
(venv) user@4189d31a5bbe:~$ LD_LIBRARY_PATH=. ./cdriver
opening mylibwithpy.so...
Initializing python...
importing numpy...
(venv) user@4189d31a5bbe:~$
**Note** that in the real build I will not have access to compiling/linking the runtime program, so this check is *NOT* a solution but seems to help diagnose the problem?
## SO... down to the actual questions
1. What is needed to, for example, get the python driver to work as expected?
2. Why do numpy's internal shared objects files not link to `libpython3.8.so`?
3. What am I missing?! :sob:
I'm hoping I'm just missing some small but crucial step in compiling the .so or configuring python.
----------
# **EDIT:**
I am able to consistently recreate the issue using a very simple docker container:
#### `Dockerfile`:
FROM ubuntu:20.04
RUN apt-get update; \
DEBIAN_FRONTEND=noninteractive apt-get install -y \
build-essential \
vim \
python3.8-dev \
python3.8-venv
RUN useradd --create-home --shell /bin/bash user
USER user
WORKDIR /home/user
RUN python3 -m venv venv
RUN /bin/bash -c "source venv/bin/activate && pip install numpy"
#### `Makefile`:
all: mylibwithpy.so cdriver
clean:
rm mylibwithpy.o mylibwithpy.so cdriver
cdriver:
gcc -L/usr/lib/x86_64-linux-gnu -Wall driver.c -o $@ -ldl
mylibwithpy.o:
gcc -L/usr/lib/x86_64-linux-gnu -I/usr/include/python3.8 -Wall -c mylibwithpy.c -o $@ -lpython3.8
mylibwithpy.so: mylibwithpy.o
gcc -L/usr/lib/x86_64-linux-gnu -Wall -fPIC -shared -Wl,-soname,$@ -o $@ mylibwithpy.o -lpython3.8
Then executing with the same command as above
LD_LIBRARY_PATH=. ./cdriver
[1]: https://docs.python.org/3/c-api/index.html
[2]: https://pubs.opengroup.org/onlinepubs/7908799/xsh/dlfcn.h.html
[3]: https://docs.python.org/3/c-api/stable.html#contents-of-limited-api
</details>
# 答案1
**得分**: 2
当使用`cdll.LoadLibrary`导出的函数时，你会在进入该方法时释放全局解释器锁（GIL）。如果你想调用Python代码，你需要重新获取锁。
例如：
```c
void someFunctionWithPython()
{
...
PyGILState_STATE state = PyGILState_Ensure();
printf("importing numpy...\n");
PyObject* numpy = PyImport_ImportModule("numpy");
if (numpy == NULL)
{
printf("Warning: error during import:\n");
PyErr_Print();
Py_Finalize();
PyGILState_Release(state);
exit(1);
}
PyObject* repr = PyObject_Repr(numpy);
PyObject* str = PyUnicode_AsEncodedString(repr, "utf-8", "~E~");
const char *bytes = PyBytes_AS_STRING(str);
printf("REPR: %s\n", bytes);
Py_XDECREF(repr);
Py_XDECREF(str);
PyGILState_Release(state);
return;
}
$ gcc $(python3.9-config --includes --ldflags --embed) -shared -o mylibwithpy.so mylibwithpy.c
$ LD_LIBRARY_PATH=. python driver.py
opening mylibwithpy.so...
.so object: <CDLL 'mylibwithpy.so', handle 1749f50 at 0x7fb603702fa0>
.so object's 'someFunctionWithPython': <_FuncPtr object at 0x7fb603679040>
calling someFunctionWithPython...
python已经初始化。
importing numpy...
REPR: <module 'numpy' from '/home/me/test/.venv/lib/python3.9/site-packages/numpy/__init__.py'>
此外，如果你查看PyDLL的文档，它说：
该类的实例的行为类似于CDLL实例，但在函数调用期间不会释放Python GIL，并在函数执行后检查Python错误标志。如果错误标志被设置，将引发Python异常。
因此，如果你在驱动程序中使用PyDLL，那么在C代码中就不需要重新获取锁：
from ctypes import PyDLL
if __name__ == "__main__":
print("opening mylibwithpy.so...");
my_so = PyDLL("mylibwithpy.so")
print(".so object: ", my_so)
print(".so object's 'someFunctionWithPython': ", my_so.someFunctionWithPython)
print("calling someFunctionWithPython...");
my_so.someFunctionWithPython()
更新：
为什么numpy的内部共享对象文件没有链接到libpython3.8.so？
我认为numpy是这样设置的，因为它期望由Python解释器调用，其中libpython已经被加载并且已经提供了符号。
也就是说，我们可以通过使用RTLD_GLOBAL来为mylibwithpy调用numpy的导入提供Python库。
对你的代码的更新很简单：
void* mylibwithpy_so = dlopen("mylibwithpy.so", RTLD_LAZY | RTLD_GLOBAL);
现在所有的Python库都会包含在内，因为它们是mylibwithpy的依赖项，这意味着当numpy加载自己的共享库时，它们将可用。
或者，你可以选择在加载mylibwithpy.so之前使用RTLD_GLOBAL加载libpythonX.Y.so，以最小化全局可用符号的数量。
printf("opening libpython3.9.so...\n");
void* libpython3_so = dlopen("libpython3.9.so", RTLD_LAZY | RTLD_GLOBAL);
if (libpython3_so == NULL){
printf("加载libpython3.9.so时发生错误：\n%s\n", dlerror());
exit(1);
}
printf("opening mylibwithpy.so...\n");
void* mylibwithpy_so = dlopen("mylibwithpy.so", RTLD_LAZY);
if (mylibwithpy_so == NULL){
printf("加载mylibwithpy.so时发生错误：\n%s\n", dlerror());
exit(1);
}
我使用以下Docker设置来重新创建和测试：
FROM ubuntu:20.04
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
build-essential \
python3.9-dev \
python3.9-venv
RUN mkdir /workspace
WORKDIR /workspace
RUN python3.9 -m venv .venv
RUN .venv/bin/python -m pip install numpy
COPY . /workspace
RUN gcc -o mylibwithpy.so mylibwithpy.c -fPIC -shared \
$(python3.9-config --includes --ldflags --embed --cflags)
RUN gcc -o cdriver driver.c -L/usr/lib/x86_64-linux-gnu -Wall -ldl
ENV LD_LIBRARY_PATH=/workspace
# 然后运行：. .venv/bin/activate && ./cdriver
英文:
When using functions exported by cdll.LoadLibrary, you're releasing the Global Interpreter Lock (GIL) as you enter the method. If you want to call python code, you need to re-acquire the lock.
e.g.
void someFunctionWithPython()
{
...
PyGILState_STATE state = PyGILState_Ensure();
printf("importing numpy...\n");
PyObject* numpy = PyImport_ImportModule("numpy");
if (numpy == NULL)
{
printf("Warning: error during import:\n");
PyErr_Print();
Py_Finalize();
PyGILState_Release(state);
exit(1);
}
PyObject* repr = PyObject_Repr(numpy);
PyObject* str = PyUnicode_AsEncodedString(repr, "utf-8", "~E~");
const char *bytes = PyBytes_AS_STRING(str);
printf("REPR: %s\n", bytes);
Py_XDECREF(repr);
Py_XDECREF(str);
PyGILState_Release(state);
return;
}
$ gcc $(python3.9-config --includes --ldflags --embed) -shared -o mylibwithpy.so mylibwithpy.c
$ LD_LIBRARY_PATH=. python driver.py
opening mylibwithpy.so...
.so object: <CDLL 'mylibwithpy.so', handle 1749f50 at 0x7fb603702fa0>
.so object's 'someFunctionWithPython': <_FuncPtr object at 0x7fb603679040>
calling someFunctionWithPython...
python alread initialized.
importing numpy...
REPR: <module 'numpy' from '/home/me/test/.venv/lib/python3.9/site-packages/numpy/__init__.py'>
Also if you look at PyDLL it says:
> Instances of this class behave like CDLL instances, except that the Python GIL is not released during the function call, and after the function execution the Python error flag is checked. If the error flag is set, a Python exception is raised.
So if you use PyDLL for your driver then you wouldn't need to re-acquire the lock in the C code:
from ctypes import PyDLL
if __name__ == "__main__":
print("opening mylibwithpy.so...");
my_so = PyDLL("mylibwithpy.so")
print(".so object: ", my_so)
print(".so object's 'someFunctionWithPython': ", my_so.someFunctionWithPython)
print("calling someFunctionWithPython...");
my_so.someFunctionWithPython()
UPDATE
> Why do numpy's internal shared objects files not link to libpython3.8.so?
I believe numpy is setup this way because it expects to be called by the python interpreter where libpython will already be loaded and have the symbols made available.
That said, we can make the python libraries available for when mylibwithpy calls the import of numpy by using RTLD_GLOBAL.
> The symbols defined by this shared object will be made available for symbol resolution of subsequently loaded shared objects.
The update to your code is simple:
void* mylibwithpy_so = dlopen("mylibwithpy.so", RTLD_LAZY | RTLD_GLOBAL);
Now all of the python libraries will be included because they are a dependency of mylibwithpy, meaning they will be available by the time that numpy loads its own shared libraries.
Alternatively, you could choose to load just libpythonX.Y.so with RTLD_GLOBAL to prior to loading mylibwithpy.so to minimize the amount symbols made globally available.
printf("opening libpython3.9.so...\n");
void* libpython3_so = dlopen("libpython3.9.so", RTLD_LAZY | RTLD_GLOBAL);
if (libpython3_so == NULL){
printf("an error occurred during loading libpython3.9.so: \n%s\n", dlerror());
exit(1);
}
printf("opening mylibwithpy.so...\n");
void* mylibwithpy_so = dlopen("mylibwithpy.so", RTLD_LAZY);
if (mylibwithpy_so == NULL){
printf("an error occurred during loading mylibwithpy.so: \n%s\n", dlerror());
exit(1);
}
Docker setup I used to recreate and test:
FROM ubuntu:20.04
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
build-essential \
python3.9-dev \
python3.9-venv
RUN mkdir /workspace
WORKDIR /workspace
RUN python3.9 -m venv .venv
RUN .venv/bin/python -m pip install numpy
COPY . /workspace
RUN gcc -o mylibwithpy.so mylibwithpy.c -fPIC -shared \
$(python3.9-config --includes --ldflags --embed --cflags)
RUN gcc -o cdriver driver.c -L/usr/lib/x86_64-linux-gnu -Wall -ldl
ENV LD_LIBRARY_PATH=/workspace
# Then run: . .venv/bin/activate && ./cdriver
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论