英文:
How is it possible that np.mean takes so much longer than looping and calculating the mean regularly?
问题
我已经构建了一个使用二元分类的神经网络，注意到它运行得非常慢。 代码如下：
predictions = self.neural_network.propagate(X_train)rounded_predictions = (predictions > 0.5).astype(int)accuracy = np.mean(rounded_predictions == y_train)
其中，x_train的形状为（16000,16），y_train的形状为（16000,）
在查找性能瓶颈时，我发现使用np.mean计算准确性平均需要0.55秒。我无法弄清楚为什么会花这么长时间，所以我尝试了另一种计算方式：
def compute_accuracy_score(y_train, predictions):"""计算网络预测的准确性分数。如果预测匹配其真实标签，则增加正确预测的计数。准确性分数是正确预测数除以总预测数。"""num_samples = len(y_train)correct_predictions = 0# 如果预测正确，则增加正确预测的计数for true_label, predicted_label in zip(y_train, predictions):if true_label == predicted_label:correct_predictions += 1# 计算准确性，即正确预测数除以样本总数accuracy = correct_predictions / num_samplesreturn accuracy
这段代码在适当的情况下运行时间低于0.01秒。
我不明白为什么与简单循环相比，np.mean需要这么长时间。
这是我编写的用于检查运行时间的代码：
def calculate_fitness(self):predictions = self.neural_network.propagate(X_train)rounded_predictions = (predictions > 0.5).astype(int)start_time = time.time()accuracy1 = compute_accuracy_score(y_train, rounded_predictions)end_time = time.time()print('使用compute accuracy计算适应度所需时间：', end_time - start_time, '秒')start_time = time.time()accuracy = np.mean(rounded_predictions == y_train)end_time = time.time()print('使用np.mean计算适应度所需时间：', end_time - start_time, '秒')print('准确性：', accuracy, '准确性1：', accuracy1)self.fitness = round(float(accuracy), 4)
准确性不同，运行时间也不同：
使用compute accuracy计算适应度所需时间：0.022114276885986328秒
使用np.mean计算适应度所需时间：0.6563560962677002秒
准确性：0.559151625 准确性1：0.6015625
使用compute accuracy计算适应度所需时间：0.021242141723632812秒
使用np.mean计算适应度所需时间：0.6553714275360107秒
准确性：0.441705 准确性1：0.398875
英文:
I've been building a neural network that uses a binary classification, and I noticed its running really slow. The code is:
predictions = self.neural_network.propagate(X_train)rounded_predictions = (predictions > 0.5).astype(int)accuracy = np.mean(rounded_predictions == y_train)
where x_train is of shape (16000,16) and y_train is of shape (16000,) <br>
When checking where the bottleneck is, I found that calculating accuracy using np.mean took an average of 0.55 seconds. I couldn't figure out what is taking it so long so i tried calculating it in a different way:
def compute_accuracy_score(y_train, predictions):"""Calculates the accuracy score of the predictions made by the network.If a prediction matches its true label, it increments the count of correct predictions.The accuracy score is the correct predictions divided by the total predictions."""num_samples = len(y_train)correct_predictions = 0# If the prediction is correct, increment the count of correct predictionsfor true_label, predicted_label in zip(y_train, predictions):if true_label == predicted_label:correct_predictions += 1# Compute accuracy as the ratio of correct predictions to total number of samplesaccuracy = correct_predictions / num_samplesreturn accuracy
This code ran it in an appropriate sub 0.01 seconds.
I don't understand why the np.mean took so much time compared to the simple loop.
This is the code I wrote to check the run time:
def calculate_fitness(self):predictions = self.neural_network.propagate(X_train)rounded_predictions = (predictions > 0.5).astype(int)start_time = time.time()accuracy1 = compute_accuracy_score(y_train, rounded_predictions)end_time = time.time()print('Time to calculate fitness using compute accuracy:', end_time - start_time, 'seconds')start_time = time.time()accuracy = np.mean(rounded_predictions == y_train)end_time = time.time()print('Time to calculate fitness using np.mean:', end_time - start_time, 'seconds')print('Accuracy:', accuracy, 'Accuracy1:', accuracy1)self.fitness = round(float(accuracy), 4)
The accuracy isnt the same, neither is the run time: <br>
Time to calculate fitness using compute accuracy: 0.022114276885986328 seconds <br>
Time to calculate fitness using np.mean: 0.6563560962677002 seconds <br>
Accuracy: 0.559151625 Accuracy1: 0.6015625 <br>
Time to calculate fitness using compute accuracy: 0.021242141723632812 seconds <br>
Time to calculate fitness using np.mean: 0.6553714275360107 seconds <br>
Accuracy: 0.441705 Accuracy1: 0.398875 <br>
答案1
得分: 1
我找到了问题。我使用的是大小为（16000,1）的nparray，它被翻译成了一个2D数组用于np.mean。然而，在使用flatten()将其变为（16000，）后，运行时间更快。
英文:
I found the problem. I was using nparray of size (16000,1) which was translated to a 2D array for np.mean. However after using flatten() to get it to (16000,) it produced faster run times.
答案2
得分: 0
你确定没有包括模型计算预测所需的时间吗？在这里：
predictions = self.neural_network.propagate(X_train)
因为我在我的笔记本电脑上使用更大的维度运行了以下代码段，但仍然需要大约4秒的处理时间。
_s = time.time()a = np.random.rand(160000000, 1)b = np.random.rand(160000000, 1)_round = (b > 0.5).astype(int)c = np.mean(_round == b)_e = time.time()
英文:
Are you sure you are not including the model's time to compute the predictions? Here:
predictions = self.neural_network.propagate(X_train)
Because I ran the following snippet on my laptop with larger dimensions and still the process time is around 4 seconds.
_s = time.time()a = np.random.rand(160000000, 1)b = np.random.rand(160000000, 1)_round = (b > 0.5).astype(int)c = np.mean(_round == b)_e = time.time()
答案3
得分: 0
你确定在第一种方法的延迟中没有考虑从模型生成预测所需的时间吗？
我尝试复制你的结果，但我看到第一种方法快了100倍。处于矢量化与非矢量化基准的预期范围内。见：
英文:
Are you sure you are not accounting the time it takes to generate the predictions from your model inside the latency of the first method?
I have tried replicating your results, but I am seeing that the first method is 100x faster. Falling in the expectation range for a vectorized vs non-vectorized benchmark. See:
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论