英文:
How to use multiprocessing Value to pass image base64 string?
问题
在我的项目中，我尝试在单独的multiprocessing.Process上运行Figure.savefig()（由于内存泄漏），但我无法为传递图像字符串分配足够长的共享变量。
参见概念示例：
```python
import base64
from multiprocessing import Process, Value
def my_func(shared_string):
with open("image.png", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
shared_string.value = encoded_string.decode()
if __name__ == "__main__":
shared_string = Value('c', b'在我的项目中，我尝试在单独的multiprocessing.Process上运行Figure.savefig()（由于内存泄漏），但我无法为传递图像字符串分配足够长的共享变量。
参见概念示例：
```python
import base64
from multiprocessing import Process, Value
def my_func(shared_string):
with open("image.png", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
shared_string.value = encoded_string.decode()
if __name__ == "__main__":
shared_string = Value('c', b'\0'*1024*1024) # 为字符串分配1MB的内存
print("Before:", shared_string.value)
p = Process(target=my_func, args=(shared_string,))
p.start()
p.join()
print("After:", shared_string.value)
'*1024*1024) # 为字符串分配1MB的内存
print("Before:", shared_string.value)
p = Process(target=my_func, args=(shared_string,))
p.start()
p.join()
print("After:", shared_string.value)
但我收到错误，可能是因为我不知道如何正确设置args
File "C:\Users\MyName\MyFiles\test.py", line 10, in <module>
shared_string = Value('c', b' File "C:\Users\MyName\MyFiles\test.py", line 10, in <module>
shared_string = Value('c', b'\0'*1024*1024) # 为字符串分配1MB的内存
File "C:\Users\Name\miniconda3\lib\multiprocessing\context.py", line 135, in Value
return Value(typecode_or_type, *args, lock=lock,
File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 74, in Value
obj = RawValue(typecode_or_type, *args)
File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 51, in RawValue
obj.__init__(*args)
TypeError: 期望一个字节、字节数组或整数字符
'*1024*1024) # 为字符串分配1MB的内存
File "C:\Users\Name\miniconda3\lib\multiprocessing\context.py", line 135, in Value
return Value(typecode_or_type, *args, lock=lock,
File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 74, in Value
obj = RawValue(typecode_or_type, *args)
File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 51, in RawValue
obj.__init__(*args)
TypeError: 期望一个字节、字节数组或整数字符
我还尝试使用multiprocessing.Array，但图像字符串的大小超过了限制。
<details>
<summary>英文:</summary>
on a project of mine I try to run Figure.savefig() on seperate multiprocessing.Process (due memory leak) but I am don't manage to allocate shared variable long enough for passing the image string.
See concept example:
import base64
from multiprocessing import Process, Value
def my_func(shared_string):
with open("image.png", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
shared_string.value = encoded_string.decode()
if name == "main":
shared_string = Value('c', b'\0'10241024) # Allocate 1MB of memory for the string
print("Before:", shared_string.value)
p = Process(target=my_func, args=(shared_string,))
p.start()
p.join()
print("After:", shared_string.value)
But I receive errors, probably because I don't understand how to set the args properly
File "C:\Users\MyName\MyFiles\test.py", line 10, in <module>
shared_string = Value('c', b'\0'10241024) # Allocate 1MB of memory for the string
File "C:\Users\Name\miniconda3\lib\multiprocessing\context.py", line 135, in Value
return Value(typecode_or_type, *args, lock=lock,
File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 74, in Value
obj = RawValue(typecode_or_type, *args)
File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 51, in RawValue
obj.init(*args)
TypeError: one character bytes, bytearray or integer expected
I also tried to use multiprocessing.Array instead, but the image string size exceed over the limit.
</details>
# 答案1
**得分**: 1
1. 一个`Value`用于存储单个数字，你应该使用`Array`。
2. 对于数组，你需要使用`array.raw`属性，而不是`array.value`。
3. 只有`bytes`数组可以存储在固定大小的数组中，Unicode字符串不能存储在`bytes`数组中，因为它没有固定的大小（一个Unicode字符可能占用多个字节）。`decode()`方法在`bytes`和`str`之间转换，这可能不是你想要做的。如果你真的需要发送Unicode字符串，最好使用`multiprocessing.Queue`。
4. 只有当你通过`http`发送数据时才需要使用`b64encode`。
以下是你的代码修改部分：
```python
import base64
from multiprocessing import Process, Value, Array
def my_func(shared_string):
with open("my_file", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
shared_string.raw = encoded_string
if __name__ == "__main__":
# 将数据写入文件，确保数据在那里
with open("my_file", 'wb') as f:
orig_string = b"1"*8
encoded_string = base64.b64decode(orig_string)
f.write(encoded_string)
shared_string = Array('c', b'
I also tried to use multiprocessing.Array instead, but the image string size exceed over the limit.
</details>
# 答案1
**得分**: 1
1. 一个`Value`用于存储单个数字，你应该使用`Array`。
2. 对于数组，你需要使用`array.raw`属性，而不是`array.value`。
3. 只有`bytes`数组可以存储在固定大小的数组中，Unicode字符串不能存储在`bytes`数组中，因为它没有固定的大小（一个Unicode字符可能占用多个字节）。`decode()`方法在`bytes`和`str`之间转换，这可能不是你想要做的。如果你真的需要发送Unicode字符串，最好使用`multiprocessing.Queue`。
4. 只有当你通过`http`发送数据时才需要使用`b64encode`。
以下是你的代码修改部分：
```python
import base64
from multiprocessing import Process, Value, Array
def my_func(shared_string):
with open("my_file", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
shared_string.raw = encoded_string
if __name__ == "__main__":
# 将数据写入文件，确保数据在那里
with open("my_file", 'wb') as f:
orig_string = b"1"*8
encoded_string = base64.b64decode(orig_string)
f.write(encoded_string)
shared_string = Array('c', b'\0' * len(orig_string)) # 为字符串分配8字节的内存空间
print("Before:", shared_string.raw)
p = Process(target=my_func, args=(shared_string,))
p.start()
p.join()
print("After:", shared_string.raw)
' * len(orig_string)) # 为字符串分配8字节的内存空间
print("Before:", shared_string.raw)
p = Process(target=my_func, args=(shared_string,))
p.start()
p.join()
print("After:", shared_string.raw)
英文:
- a Value is for storing a single number, you are looking for an
Arrayinstead. - for arrays you need to use the
array.rawattribute instead ofarray.value. - only
bytesarrays can be stored in a fixed size array, a unicode string cannot be stored in a bytes array because it doesn't have a fixed size (a unicode character can take up more than one byte),decode()converts betweenbytesandstr, this is probably not what you want to do, if you really must send a unicode string you are better off usingmultiprocessing.Queue. - just note you only need
b64encodeif you are sending it overhttp.
the following modification to your code shows how it can be done.
import base64
from multiprocessing import Process, Value, Array
def my_func(shared_string):
with open("my_file", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
shared_string.raw = encoded_string
if __name__ == "__main__":
# write data to file, to make sure it is there
with open("my_file",'wb') as f:
orig_string = b"1"*8
encoded_string = base64.b64decode(orig_string)
f.write(encoded_string)
shared_string = Array('c', b' import base64
from multiprocessing import Process, Value, Array
def my_func(shared_string):
with open("my_file", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
shared_string.raw = encoded_string
if __name__ == "__main__":
# write data to file, to make sure it is there
with open("my_file",'wb') as f:
orig_string = b"1"*8
encoded_string = base64.b64decode(orig_string)
f.write(encoded_string)
shared_string = Array('c', b'\0' * len(orig_string)) # Allocate 8 bytes of memory for the string
print("Before:", shared_string.raw)
p = Process(target=my_func, args=(shared_string,))
p.start()
p.join()
print("After:", shared_string.raw)
' * len(orig_string)) # Allocate 8 bytes of memory for the string
print("Before:", shared_string.raw)
p = Process(target=my_func, args=(shared_string,))
p.start()
p.join()
print("After:", shared_string.raw)
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论