英文:
memory increase after upgrading setuptools
问题
升级 setuptools 到版本 65.3.0 后，突然间系统中大多数进程的内存占用增加了。为了找出问题，我尝试查找有问题的 import，并检查了一下，看起来脚本中的 distutils 导入导致了新版本中加载了大量动态库，导致加载时间更长，内存使用增加。
from distutils.version import StrictVersion
import time
while True:
time.sleep(1000)
[root@controller(FHGW-83) /home/robot]
# pmap 2440334 | wc -l
196
[root@controller(FHGW-83) /home/robot]
# pmap 2440334 | grep cpython | wc -l
111
# pmap 2432196 | grep cpython
00007f8606e8a000 72K r---- _ssl.cpython-39-x86_64-linux-gnu.so
...
00007f8606f02000 24K r-x-- _json.cpython-39-x86_64-linux-gnu.so
...
00007f8607296000 4K rw--- _csv.cpython-39-x86_64-linux-gnu.so
...
00007f860786f000 4K rw--- _sha512.cpython-39-x86_64-linux-gnu.so
...
00007f86078fe000 8K rw--- pyexpat.cpython-39-x86_64-linux-gnu.so
...
00007f86079f9000 4K rw--- _struct.cpython-39-x86_64-linux-gnu.so
...
等等...
$ top -p 2440334
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
2440334 root 20 0 31.9m 27.2m 0.0 0.2 0:00.68 S python test.py
在 setuptools 版本为 57.0.0 的机器上，不会出现此问题。虚拟内存和实际内存占用也较低。
# python test.py &
[1] 1562394
]
# pmap 1562394 | wc -l
67
# pmap 1562394 | grep cpyth | wc -l
5
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
1562394 root 20 0 14.0m 9.3m 0.0 0.1 0:00.05 S python test.py
这里与 setuptools 有什么关系？Python 版本（3.9.16）在两台机器上都相同。为什么这个脚本会导致Python加载大量库？
英文:
After upgrading setuptools to 65.3.0 all of sudden there is memory increase for majority of the process in the system. To narrow down the issue, I tried to find out the problematic import and checked, it looks like the distutils import in the script is causing loading of lot of dynamic libraries in the new version and causing more time to load and also increased memory usage.
from distutils.version import StrictVersion
import time
while True:
time.sleep(1000)
[root@controller(FHGW-83) /home/robot]
# pmap 2440334 | wc -l
196
[root@controller(FHGW-83) /home/robot]
# pmap 2440334 | grep cpython | wc -l
111
# pmap 2432196 | grep cpython
00007f8606e8a000 72K r---- _ssl.cpython-39-x86_64-linux-gnu.so
...
00007f8606f02000 24K r-x-- _json.cpython-39-x86_64-linux-gnu.so
...
00007f8607296000 4K rw--- _csv.cpython-39-x86_64-linux-gnu.so
...
00007f860786f000 4K rw--- _sha512.cpython-39-x86_64-linux-gnu.so
...
00007f86078fe000 8K rw--- pyexpat.cpython-39-x86_64-linux-gnu.so
...
00007f86079f9000 4K rw--- _struct.cpython-39-x86_64-linux-gnu.so
...
and more...
$ top -p 2440334
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
2440334 root 20 0 31.9m 27.2m 0.0 0.2 0:00.68 S python test.py
In a machine where setuptools is at 57.0.0, this issue is not seen.
virtual & RES memory consumption is also less.
# python test.py &
[1] 1562394
]
# pmap 1562394 | wc -l
67
# pmap 1562394 | grep cpyth | wc -l
5
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
1562394 root 20 0 14.0m 9.3m 0.0 0.1 0:00.05 S python test.py
Here what is the relation with setuptools? python version (3.9.16) is same in both the machines. why is the script causing python to load lot of libraries?
答案1
得分: 0
我能够在Python 3.9中解决这个问题，只需导出 SETUPTOOLS_USE_DISTUTILS=stdlib 即可。
这将确保使用标准Python分发中的 distutils。如果我们导出 SETUPTOOLS_USE_DISTUTILS=local，则会使用setuptools包中的distutils。
>>> import distutils
>>> distutils.__file__
'/usr/lib/python3.9/distutils/__init__.py'
# 在升级setuptools后
>>> import distutils
>>> distutils.__file__
'/usr/lib/python3.9/site-packages/setuptools/_distutils/__init__.py'
英文:
I was able to solve this in python 3.9 by simply exporting SETUPTOOLS_USE_DISTUTILS=stdlib
This will make sure to use distutils from the standard python distribution. If we export SETUPTOOLS_USE_DISTUTILS=local then distutils is used from setuptools package.
>>> import distutils
>>> distutils.__file__
'/usr/lib/python3.9/distutils/__init__.py'
#after setuptools upgrade
>>> import distutils
>>> distutils.__file__
'/usr/lib/python3.9/site-packages/setuptools/_distutils/__init__.py'
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论