加速 np.sum 使用多进程

huangapple
go评论89阅读模式
英文:

Speeding Up np.sum With multiprocessing

问题

如果我有一个大小在10^8到10^9之间的NumPy数组，是否有可能比np.sum更快地计算其总和？

我尝试过使用multiprocessing和fork，但无论工作进程的数量（1-4）如何，似乎都比只调用np.sum慢。我在一台配备2 GHz双核英特尔Core i5处理器的Mac上使用Python 3.8。不确定如果我有更多的CPU核心是否会有不同的结果。

我的代码：

  1. import concurrent.futures
  2. import multiprocessing as mp
  3. import time
  4. from concurrent.futures.process import ProcessPoolExecutor
  6. import numpy as np
  8. # 基于：https://luis-sena.medium.com/sharing-big-numpy-arrays-across-python-processes-abf0dc2a0ab2
  10. def np_sum_global(start, stop):
  11.     return np.sum(data[start:stop])
  13. def benchmark():
  14.     st = time.time()
  15.     ARRAY_SIZE = int(3e8)
  16.     print("数组大小 =", ARRAY_SIZE)
  17.     global data
  18.     data = np.random.random(ARRAY_SIZE)
  19.     print("生成数组时间 =", time.time() - st)
  20.     print("CPU核心数 =", mp.cpu_count())
  22.     for trial in range(5):
  23.         print("试验 =", trial)
  24.         st = time.time()
  25.         s = np.sum(data)
  26.         print("方法1", time.time() - st, s)
  28.         for NUM_WORKERS in range(1, 5):
  29.             st = time.time()
  30.             futures = []
  31.             with ProcessPoolExecutor(max_workers=NUM_WORKERS) as executor:
  32.                 for i in range(0, NUM_WORKERS):
  33.                     futures.append(
  34.                         executor.submit(
  35.                             np_sum_global,
  36.                             ARRAY_SIZE * i // NUM_WORKERS,
  37.                             ARRAY_SIZE * (i + 1) // NUM_WORKERS,
  38.                         )
  39.                     )
  40.             futures, _ = concurrent.futures.wait(futures)
  41.             s = sum(future.result() for future in futures)
  42.             print("工作进程数 =", NUM_WORKERS, time.time() - st, s)
  43.         print()
  45. if __name__ == "__main__":
  46.     mp.set_start_method("fork")
  47.     benchmark()

输出：

  1. 数组大小 = 300000000
  2. 生成数组时间 5.1455769538879395
  3. CPU核心数 = 4
  4. 试验 = 0
  5. 方法1 0.29593801498413086 150004049.39847052
  6. 工作进程数 = 1 1.8904719352722168 150004049.39847052
  7. 工作进程数 = 2 1.2082111835479736 150004049.39847034
  8. 工作进程数 = 3 1.2650330066680908 150004049.39847082
  9. 工作进程数 = 4 1.233708143234253 150004049.39847046
  11. 试验 = 1
  12. 方法1 0.5861320495605469 150004049.39847052
  13. 工作进程数 = 1 1.801928997039795 150004049.39847052
  14. 工作进程数 = 2 1.165492057800293 150004049.39847034
  15. 工作进程数 = 3 1.2669389247894287 150004049.39847082
  16. 工作进程数 = 4 1.2941789627075195 150004049.39847043
  18. 试验 = 2
  19. 方法1 0.44912219047546387 150004049.39847052
  20. 工作进程数 = 1 1.8038971424102783 150004049.39847052
  21. 工作进程数 = 2 1.1491520404815674 150004049.39847034
  22. 工作进程数 = 3 1.3324410915374756 150004049.39847082
  23. 工作进程数 = 4 1.4198641777038574 150004049.39847046
  25. 试验 = 3
  26. 方法1 0.5163640975952148 150004049.39847052
  27. 工作进程数 = 1 3.248213052749634 150004049.39847052
  28. 工作进程数 = 2 2.5148861408233643 150004049.39847034
  29. 工作进程数 = 3 1.0224149227142334 150004049.39847082
  30. 工作进程数 = 4 1.20924711227417 150004049.39847046
  32. 试验 = 4
  33. 方法1 1.2363107204437256 150004049.39847052
  34. 工作进程数 = 1 1.8627309799194336 150004049.39847052
  35. 工作进程数 = 2 1.233341932296753 150004049.39847034
  36. 工作进程数 = 3 1.3235111236572266 150004049.39847082
  37. 工作进程数 = 4 1.344843864440918 150004049.39847046

我查阅了一些链接：

英文:

If I have a numpy array of size 10^8 to 10^9, is it possible to compute its sum faster than np.sum?

I've tried using multiprocessing with fork, but it seems to be slower than just calling np.sum, regardless of the number of workers (1-4). I'm using Python 3.8 on a Mac with a 2 GHz Dual-Core Intel Core i5 processor. Not sure whether the results would be different if I had more CPUs.

My code:

  1. import concurrent.futures
  2. import multiprocessing as mp
  3. import time
  4. from concurrent.futures.process import ProcessPoolExecutor
  6. import numpy as np
  8. # based on: https://luis-sena.medium.com/sharing-big-numpy-arrays-across-python-processes-abf0dc2a0ab2
  11. def np_sum_global(start, stop):
  12.     return np.sum(data[start:stop])
  15. def benchmark():
  16.     st = time.time()
  17.     ARRAY_SIZE = int(3e8)
  18.     print("array size =", ARRAY_SIZE)
  19.     global data
  20.     data = np.random.random(ARRAY_SIZE)
  21.     print("generated", time.time() - st)
  22.     print("CPU Count =", mp.cpu_count())
  24.     for trial in range(5):
  25.         print("TRIAL =", trial)
  26.         st = time.time()
  27.         s = np.sum(data)
  28.         print("method 1", time.time() - st, s)
  30.         for NUM_WORKERS in range(1, 5):
  31.             st = time.time()
  32.             futures = []
  33.             with ProcessPoolExecutor(max_workers=NUM_WORKERS) as executor:
  34.                 for i in range(0, NUM_WORKERS):
  35.                     futures.append(
  36.                         executor.submit(
  37.                             np_sum_global,
  38.                             ARRAY_SIZE * i // NUM_WORKERS,
  39.                             ARRAY_SIZE * (i + 1) // NUM_WORKERS,
  40.                         )
  41.                     )
  42.             futures, _ = concurrent.futures.wait(futures)
  43.             s = sum(future.result() for future in futures)
  44.             print("workers =", NUM_WORKERS, time.time() - st, s)
  45.         print()
  48. if __name__ == "__main__":
  49.     mp.set_start_method("fork")
  50.     benchmark()

Output:

  1. array size = 300000000
  2. generated 5.1455769538879395
  3. CPU Count = 4
  4. TRIAL = 0
  5. method 1 0.29593801498413086 150004049.39847052
  6. workers = 1 1.8904719352722168 150004049.39847052
  7. workers = 2 1.2082111835479736 150004049.39847034
  8. workers = 3 1.2650330066680908 150004049.39847082
  9. workers = 4 1.233708143234253 150004049.39847046
  11. TRIAL = 1
  12. method 1 0.5861320495605469 150004049.39847052
  13. workers = 1 1.801928997039795 150004049.39847052
  14. workers = 2 1.165492057800293 150004049.39847034
  15. workers = 3 1.2669389247894287 150004049.39847082
  16. workers = 4 1.2941789627075195 150004049.39847043
  18. TRIAL = 2
  19. method 1 0.44912219047546387 150004049.39847052
  20. workers = 1 1.8038971424102783 150004049.39847052
  21. workers = 2 1.1491520404815674 150004049.39847034
  22. workers = 3 1.3324410915374756 150004049.39847082
  23. workers = 4 1.4198641777038574 150004049.39847046
  25. TRIAL = 3
  26. method 1 0.5163640975952148 150004049.39847052
  27. workers = 1 3.248213052749634 150004049.39847052
  28. workers = 2 2.5148861408233643 150004049.39847034
  29. workers = 3 1.0224149227142334 150004049.39847082
  30. workers = 4 1.20924711227417 150004049.39847046
  32. TRIAL = 4
  33. method 1 1.2363107204437256 150004049.39847052
  34. workers = 1 1.8627309799194336 150004049.39847052
  35. workers = 2 1.233341932296753 150004049.39847034
  36. workers = 3 1.3235111236572266 150004049.39847082
  37. workers = 4 1.344843864440918 150004049.39847046

Some links I've looked at:

答案1

得分: 1

这是一个使用Numba进行性能基准测试的示例。它首先需要编译代码，这会导致第一次运行速度较慢。后续运行通常比NumPy快两到三倍。因此，是否值得使用Numba取决于你运行代码的频率。

  1. import numba
  2. import numpy as np
  3. import time
  5. # 基于：https://luis-sena.medium.com/sharing-big-numpy-arrays-across-python-processes-abf0dc2a0ab2
  7. @numba.jit(nopython=True, parallel=True, cache=True)
  8. def numba_sum(data):
  9.     return np.sum(data)
  11. def benchmark():
  12.     st = time.time()
  13.     ARRAY_SIZE = int(3e8)
  14.     print("数组大小 =", ARRAY_SIZE)
  15.     global data
  16.     data = np.random.random(ARRAY_SIZE)
  17.     print("生成了", time.time() - st)
  19.     for trial in range(5):
  20.         print("试验 =", trial)
  21.         st = time.time()
  22.         s = np.sum(data)
  23.         print("方法 1", time.time() - st, s)
  24.         print("试验 =", trial)
  25.         st = time.time()
  26.         s = numba_sum(data)
  27.         print("方法 2", time.time() - st, s)
  29. if __name__ == "__main__":
  30.     benchmark()

希望这对你有帮助。

英文:

Here's a benchmark with numba. It first has to compile the code which makes the first run a lot slower. The next runs are about twice to 3 times faster than numpy. So it depends on how often you run the code, if numba is worth it for you or not.

  1. import numba
  2. import numpy as np
  3. import time
  5. # based on: https://luis-sena.medium.com/sharing-big-numpy-arrays-across-python-processes-abf0dc2a0ab2
  7. @numba.jit(nopython=True, parallel=True, cache=True)
  8. def numba_sum(data):
  9.     return np.sum(data)
  11. def benchmark():
  12.     st = time.time()
  13.     ARRAY_SIZE = int(3e8)
  14.     print("array size =", ARRAY_SIZE)
  15.     global data
  16.     data = np.random.random(ARRAY_SIZE)
  17.     print("generated", time.time() - st)
  19.     for trial in range(5):
  20.         print("TRIAL =", trial)
  21.         st = time.time()
  22.         s = np.sum(data)
  23.         print("method 1", time.time() - st, s)
  24.         print("TRIAL =", trial)
  25.         st = time.time()
  26.         s = numba_sum(data)
  27.         print("method 2", time.time() - st, s)
  30. if __name__ == "__main__":
  31.     benchmark()

huangapple
  • 本文由 发表于 2023年4月4日 09:35:40
  • 转载请务必保留本文链接：https://go.coder-hub.com/75924868.html
  • numpy
  • python-3.x
  • python-multiprocessing
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定