如何使用多进程的Value来传递图像的base64字符串?

huangapple go评论71阅读模式
英文:

How to use multiprocessing Value to pass image base64 string?

问题

在我的项目中我尝试在单独的multiprocessing.Process上运行Figure.savefig()由于内存泄漏),但我无法为传递图像字符串分配足够长的共享变量

参见概念示例

```python
import base64
from multiprocessing import Process, Value

def my_func(shared_string):
    with open("image.png", "rb") as image_file:
       
        encoded_string = base64.b64encode(image_file.read())
        shared_string.value = encoded_string.decode()

if __name__ == "__main__":
    shared_string = Value('c', b'
在我的项目中我尝试在单独的multiprocessing.Process上运行Figure.savefig()由于内存泄漏),但我无法为传递图像字符串分配足够长的共享变量

参见概念示例

```python
import base64
from multiprocessing import Process, Value

def my_func(shared_string):
    with open("image.png", "rb") as image_file:
       
        encoded_string = base64.b64encode(image_file.read())
        shared_string.value = encoded_string.decode()

if __name__ == "__main__":
    shared_string = Value('c', b'\0'*1024*1024) # 为字符串分配1MB的内存
    print("Before:", shared_string.value)

    p = Process(target=my_func, args=(shared_string,))
    p.start()
    p.join()

    print("After:", shared_string.value)
'*1024*1024) # 为字符串分配1MB的内存
print("Before:", shared_string.value) p = Process(target=my_func, args=(shared_string,)) p.start() p.join() print("After:", shared_string.value)

但我收到错误,可能是因为我不知道如何正确设置args

  File "C:\Users\MyName\MyFiles\test.py", line 10, in <module>
    shared_string = Value('c', b'
  File "C:\Users\MyName\MyFiles\test.py", line 10, in <module>
shared_string = Value('c', b'\0'*1024*1024) # 为字符串分配1MB的内存
File "C:\Users\Name\miniconda3\lib\multiprocessing\context.py", line 135, in Value
return Value(typecode_or_type, *args, lock=lock,
File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 74, in Value
obj = RawValue(typecode_or_type, *args)
File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 51, in RawValue
obj.__init__(*args)
TypeError: 期望一个字节、字节数组或整数字符
'*1024*1024) # 为字符串分配1MB的内存 File "C:\Users\Name\miniconda3\lib\multiprocessing\context.py", line 135, in Value return Value(typecode_or_type, *args, lock=lock, File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 74, in Value obj = RawValue(typecode_or_type, *args) File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 51, in RawValue obj.__init__(*args) TypeError: 期望一个字节、字节数组或整数字符

我还尝试使用multiprocessing.Array,但图像字符串的大小超过了限制。


<details>
<summary>英文:</summary>

on a project of mine I try to run Figure.savefig() on seperate multiprocessing.Process (due memory leak) but I am don&#39;t manage to allocate shared variable long enough for passing the image string.

See concept example:

import base64
from multiprocessing import Process, Value

def my_func(shared_string):
with open("image.png", "rb") as image_file:

    encoded_string = base64.b64encode(image_file.read())
    shared_string.value = encoded_string.decode()

if name == "main":
shared_string = Value('c', b'\0'10241024) # Allocate 1MB of memory for the string
print("Before:", shared_string.value)

p = Process(target=my_func, args=(shared_string,))
p.start()
p.join()

print(&quot;After:&quot;, shared_string.value)

But I receive errors, probably because I don&#39;t understand how to set the args properly 

File "C:\Users\MyName\MyFiles\test.py", line 10, in <module>
shared_string = Value('c', b'\0'10241024) # Allocate 1MB of memory for the string
File "C:\Users\Name\miniconda3\lib\multiprocessing\context.py", line 135, in Value
return Value(typecode_or_type, *args, lock=lock,
File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 74, in Value
obj = RawValue(typecode_or_type, *args)
File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 51, in RawValue
obj.init(*args)
TypeError: one character bytes, bytearray or integer expected




I also tried to use multiprocessing.Array instead, but the image string size exceed over the limit.


</details>


# 答案1
**得分**: 1

1. 一个`Value`用于存储单个数字,你应该使用`Array`。

2. 对于数组,你需要使用`array.raw`属性,而不是`array.value`。

3. 只有`bytes`数组可以存储在固定大小的数组中,Unicode字符串不能存储在`bytes`数组中,因为它没有固定的大小(一个Unicode字符可能占用多个字节)。`decode()`方法在`bytes`和`str`之间转换,这可能不是你想要做的。如果你真的需要发送Unicode字符串,最好使用`multiprocessing.Queue`。

4. 只有当你通过`http`发送数据时才需要使用`b64encode`。

以下是你的代码修改部分:

```python
import base64
from multiprocessing import Process, Value, Array

def my_func(shared_string):
    with open("my_file", "rb") as image_file:
        encoded_string = base64.b64encode(image_file.read())
        shared_string.raw = encoded_string

if __name__ == "__main__":
    # 将数据写入文件,确保数据在那里
    with open("my_file", 'wb') as f:
        orig_string = b"1"*8
        encoded_string = base64.b64decode(orig_string)
        f.write(encoded_string)

    shared_string = Array('c', b'

I also tried to use multiprocessing.Array instead, but the image string size exceed over the limit.
</details>
# 答案1
**得分**: 1
1. 一个`Value`用于存储单个数字,你应该使用`Array`。
2. 对于数组,你需要使用`array.raw`属性,而不是`array.value`。
3. 只有`bytes`数组可以存储在固定大小的数组中,Unicode字符串不能存储在`bytes`数组中,因为它没有固定的大小(一个Unicode字符可能占用多个字节)。`decode()`方法在`bytes`和`str`之间转换,这可能不是你想要做的。如果你真的需要发送Unicode字符串,最好使用`multiprocessing.Queue`。
4. 只有当你通过`http`发送数据时才需要使用`b64encode`。
以下是你的代码修改部分:
```python
import base64
from multiprocessing import Process, Value, Array
def my_func(shared_string):
with open("my_file", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
shared_string.raw = encoded_string
if __name__ == "__main__":
# 将数据写入文件,确保数据在那里
with open("my_file", 'wb') as f:
orig_string = b"1"*8
encoded_string = base64.b64decode(orig_string)
f.write(encoded_string)
shared_string = Array('c', b'\0' * len(orig_string))  # 为字符串分配8字节的内存空间
print("Before:", shared_string.raw)
p = Process(target=my_func, args=(shared_string,))
p.start()
p.join()
print("After:", shared_string.raw)
' * len(orig_string)) # 为字符串分配8字节的内存空间 print("Before:", shared_string.raw) p = Process(target=my_func, args=(shared_string,)) p.start() p.join() print("After:", shared_string.raw)
英文:
  1. a Value is for storing a single number, you are looking for an Array instead.
  2. for arrays you need to use the array.raw attribute instead of array.value.
  3. only bytes arrays can be stored in a fixed size array, a unicode string cannot be stored in a bytes array because it doesn't have a fixed size (a unicode character can take up more than one byte), decode() converts between bytes and str, this is probably not what you want to do, if you really must send a unicode string you are better off using multiprocessing.Queue.
  4. just note you only need b64encode if you are sending it over http.

the following modification to your code shows how it can be done.

 import base64
from multiprocessing import Process, Value, Array


def my_func(shared_string):
    with open(&quot;my_file&quot;, &quot;rb&quot;) as image_file:
        encoded_string = base64.b64encode(image_file.read())
        shared_string.raw = encoded_string


if __name__ == &quot;__main__&quot;:

    # write data to file, to make sure it is there
    with open(&quot;my_file&quot;,&#39;wb&#39;) as f:
        orig_string = b&quot;1&quot;*8
        encoded_string = base64.b64decode(orig_string)
        f.write(encoded_string)

    shared_string = Array(&#39;c&#39;, b&#39;
 import base64
from multiprocessing import Process, Value, Array


def my_func(shared_string):
    with open(&quot;my_file&quot;, &quot;rb&quot;) as image_file:
        encoded_string = base64.b64encode(image_file.read())
        shared_string.raw = encoded_string


if __name__ == &quot;__main__&quot;:

    # write data to file, to make sure it is there
    with open(&quot;my_file&quot;,&#39;wb&#39;) as f:
        orig_string = b&quot;1&quot;*8
        encoded_string = base64.b64decode(orig_string)
        f.write(encoded_string)

    shared_string = Array(&#39;c&#39;, b&#39;\0&#39; * len(orig_string))  # Allocate 8 bytes of memory for the string
    print(&quot;Before:&quot;, shared_string.raw)

    p = Process(target=my_func, args=(shared_string,))
    p.start()
    p.join()

    print(&quot;After:&quot;, shared_string.raw)
&#39; * len(orig_string)) # Allocate 8 bytes of memory for the string
print(&quot;Before:&quot;, shared_string.raw) p = Process(target=my_func, args=(shared_string,)) p.start() p.join() print(&quot;After:&quot;, shared_string.raw)

huangapple
  • 本文由 发表于 2023年3月7日 16:19:16
  • 转载请务必保留本文链接:https://go.coder-hub.com/75659468.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定