英文:
Summing large matrices of image data efficiently
问题
如何最高效地对大型 (100, 1024, 1024) .npz 文件进行求和？是否有更好的文件格式来存储给定维度的Python数据？
我目前使用以下方法来对矩阵求和：
summed_matrix = np.sum([np.load(file) for file in files_list])
这超出了我的可用内存。
英文:
How do I sum large (100, 1024, 1024) .npz files most efficiently? Is there a better file format to store python data of the given dimensions?
I currently use this to sum the matrices:
summed_matrix = np.sum([np.load(file) for file in files_list])
This exceeds my available memory.
答案1
得分: 7
目前，您一次加载所有文件，然后才开始求和。如果内存是一个问题，更好的方法是逐个读取并相加：
summed_matrix = np.zeros((100, 1024, 1024))
for file in files_list:
summed_matrix += np.load(file)
英文:
Currently, you're loading all files at once and only then begin summing. If memory is a concern, a better approach would be to read and add one at a time:
summed_matrix = np.zeros((100, 1024, 1024))
for file in files_list:
summed_matrix += np.load(file)
答案2
得分: 5
你可以尝试使用 functools.reduce 和 numpy.add 的生成器：
from functools import reduce
summed_matrix = reduce(np.add, (np.load(file) for file in files_list))
# 或者函数式方式
summed_matrix = reduce(np.add, map(np.load, files_list))
英文:
You could try a generator with functools.reduce and numpy.add:
from functools import reduce
summed_matrix = reduce(np.add, (np.load(file) for file in files_list))
# or functional-style
summed_matrix = reduce(np.add, map(np.load, files_list))
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论