英文:
Which axis to use to normalise 3 dimensional data?
问题
我想知道在三维数据上进行归一化时，最好使用哪个轴（默认是最后一个轴）。例如，我有一个形状为 3 x 4 x 5（批量大小 x 元素数量 x 特征数量）的张量。在这种情况下，我应该使用轴 1 来对每个特征进行归一化，还是使用默认的轴 -1（2）？:
data = tf.constant(np.arange(3 * 4 * 5).reshape(3, 4, 5) * 10, dtype=tf.float32)
print(data)
tf.Tensor(
[[[ 0. 10. 20. 30. 40.]
[ 50. 60. 70. 80. 90.]
[100. 110. 120. 130. 140.]
[150. 160. 170. 180. 190.]]
[[200. 210. 220. 230. 240.]
[250. 260. 270. 280. 290.]
[300. 310. 320. 330. 340.]
[350. 360. 370. 380. 390.]]
[[400. 410. 420. 430. 440.]
[450. 460. 470. 480. 490.]
[500. 510. 520. 530. 540.]
[550. 560. 570. 580. 590.]]], shape=(3, 4, 5), dtype=float32)
layer = tf.keras.layers.LayerNormalization(axis=1)
output = layer(data)
print(output)
tf.Tensor(
[[[-1.3416405 -1.3416405 -1.3416406 -1.3416405 -1.3416405 ]
[-0.44721347 -0.44721353 -0.44721353 -0.44721353 -0.4472134 ]
[ 0.44721353 0.44721353 0.4472134 0.44721353 0.44721365]
[ 1.3416405 1.3416405 1.3416405 1.3416406 1.3416407 ]]
[[-1.3416407 -1.3416407 -1.3416407 -1.3416405 -1.3416405 ]
[-0.44721365 -0.44721365 -0.44721365 -0.44721317 -0.44721317]
[ 0.44721317 0.44721317 0.44721317 0.44721365 0.44721365]
[ 1.3416405 1.3416405 1.3416405 1.341641 1.3416405 ]]
[[-1.341641 -1.341641 -1.341641 -1.341641 -1.34164 ]
[-0.44721413 -0.44721413 -0.44721413 -0.44721413 -0.44721317]
[ 0.44721317 0.44721317 0.44721317 0.44721317 0.44721413]
[ 1.3416405 1.3416405 1.3416405 1.3416405 1.3416414 ]]], shape=(3, 4, 5), dtype=float32)
英文:
I want to know which axis is better to use for normalisation for a 3 dimensional data (it is the last axis by default). For example, I have this tensor with the shape 3 x 4 x 5 (batch_size x number_of_elements x number_of_features). Should I use axis 1 in this case so that I normalise each feature or the default axis -1 (2)?:
data = tf.constant(np.arange(3 * 4 * 5).reshape(3, 4, 5) * 10, dtype=tf.float32)
print(data)
tf.Tensor(
[[[ 0. 10. 20. 30. 40.]
[ 50. 60. 70. 80. 90.]
[100. 110. 120. 130. 140.]
[150. 160. 170. 180. 190.]]
[[200. 210. 220. 230. 240.]
[250. 260. 270. 280. 290.]
[300. 310. 320. 330. 340.]
[350. 360. 370. 380. 390.]]
[[400. 410. 420. 430. 440.]
[450. 460. 470. 480. 490.]
[500. 510. 520. 530. 540.]
[550. 560. 570. 580. 590.]]], shape=(3, 4, 5), dtype=float32)
layer = tf.keras.layers.LayerNormalization(axis=1)
output = layer(data)
print(output)
tf.Tensor(
[[[-1.3416405 -1.3416405 -1.3416406 -1.3416405 -1.3416405 ]
[-0.44721347 -0.44721353 -0.44721353 -0.44721353 -0.4472134 ]
[ 0.44721353 0.44721353 0.4472134 0.44721353 0.44721365]
[ 1.3416405 1.3416405 1.3416405 1.3416406 1.3416407 ]]
[[-1.3416407 -1.3416407 -1.3416407 -1.3416405 -1.3416405 ]
[-0.44721365 -0.44721365 -0.44721365 -0.44721317 -0.44721317]
[ 0.44721317 0.44721317 0.44721317 0.44721365 0.44721365]
[ 1.3416405 1.3416405 1.3416405 1.341641 1.3416405 ]]
[[-1.341641 -1.341641 -1.341641 -1.341641 -1.34164 ]
[-0.44721413 -0.44721413 -0.44721413 -0.44721413 -0.44721317]
[ 0.44721317 0.44721317 0.44721317 0.44721317 0.44721413]
[ 1.3416405 1.3416405 1.3416405 1.3416405 1.3416414 ]]], shape=(3, 4, 5), dtype=float32)
答案1
得分: 2
层归一化是在特征轴/轴上执行的。我不确定你的 axis=1 是什么意思，但由于你有多维数据，我假设这是一个（空间？）特征。所以在你的情况下，你应该使用将列表传递给函数的选项
tf.keras.layers.LayerNormalization(axis=[1, 2])
一些背景信息可以在这里找到，其中提到：
> [...] 假设我们有一个表示（B，H，W，C）的4D张量，表示图像的批次、高度、宽度和通道。
> 批次归一化 → 对通道（1,1,1,c）求均值和方差
> 层归一化 → 对批次（b,1,1,1）求均值和方差
> 实例归一化 → 对批次/通道（b,1,1,c）求均值和方差
英文:
Layer Normalization is performed across feature axis/axes. I am not sure what your axis=1 is, but, since you have multidimensional data, I assume this is a (spatial?) feature. So in your case you should use the option to pass a list to the function
tf.keras.layers.LayerNormalization(axis=[1, 2])
some background can be found here where it says:
> [...] Assuming we have 4D tensor that represents (B,H,W,C), which
> stands for Batch, Height, Width, and Channel of image.
> Batch Norm → Take mean and variance respect to channel (1,1,1,c)
> Layer Norm → Take mean and variance respect to batch (b,1,1,1)
> Instance Norm → Take mean and variance respect to batch/channel (b,1,1,c)
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论