英文:
How to reproduce the default behaviour of tf.get_variable with numpy? (TensorFlow v1.15.0)
问题
使用tf.get_variable
初始化的变量默认情况下是从Glorot/Xavier分布中采样的。
如何使用numpy.random
复现该分布?
这可能归结为设置正确的随机种子 - 但是,我尝试了一下(请参见下面的最小工作示例),但没有成功。有人知道如何使下面示例中的模式输出相同,或者理解它们为什么不同吗?
import tensorflow as tf
import numpy as np
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--mode', type=int)
args = parser.parse_args()
fan_in = 2
fan_out = 3
glorot_scale = 1. / max(1., (fan_in + fan_out) / 2.)
glorot_limit = limit = np.sqrt(3.0 * glorot_scale)
mode = args.mode
tf.set_random_seed(0)
if mode in [4,5]:
np.random.seed(0)
print(mode)
if mode == 0:
variable = tf.compat.v1.get_variable(name='variable', shape=[fan_in, fan_out])
elif mode == 1:
variable = tf.compat.v1.get_variable(name='variable', shape=[fan_in, fan_out],
initializer=tf.glorot_uniform_initializer(seed=0))
elif mode == 2:
variable = tf.Variable(tf.glorot_uniform_initializer(seed=0)(
shape=[fan_in, fan_out], dtype=tf.float32))
elif mode == 3:
variable = tf.random.stateless_uniform(
shape=(fan_in,fan_out), seed=[0, 0],
minval=-glorot_limit, maxval=glorot_limit)
elif mode == 4:
variable = tf.Variable(
np.random.uniform(-glorot_limit, glorot_limit, (fan_in, fan_out)),
dtype=tf.float32)
elif mode == 5:
variable = np.array(np.random.uniform(-glorot_limit, glorot_limit, (fan_in, fan_out)),
dtype=np.float32)
if mode in range(5):
sess = tf.compat.v1.Session()
sess.run(tf.compat.v1.global_variables_initializer())
variable = sess.run(variable)
print(variable)
输出如下:
0
[[-0.39295787 0.4208691 0.53050697]
[ 0.8091326 1.007618 0.95607924]]
1
[[-1.0521597 -1.0800165 -0.6794561 ]
[ 0.60745895 -0.17927146 0.5341264 ]]
2
[[-1.0521597 -1.0800165 -0.6794561 ]
[ 0.60745895 -0.17927146 0.5341264 ]]
3
[[-0.6331334 0.45134532 0.657406 ]
[ 0.59327614 -0.24409002 0.81141686]]
4
[[ 0.10694503 0.4714563 0.22514328]
[ 0.09833413 -0.16726395 0.31963798]]
5
[[ 0.10694503 0.4714563 0.22514328]
[ 0.09833413 -0.16726395 0.31963798]]
英文:
Variables initialized with tf.get_variable
are (to my knowledge), per default, sampled from a Glorot/Xavier distribution.
How can one reproduce that distribution with numpy.random
?
It probably boils down to setting the correct random seeds - which, however, I do not succeed in (see the following minimal working example). Does someone know how to make the outputs of the modes in the example below the same or understand why they differ?
import tensorflow as tf
import numpy as np
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--mode', type=int)
args = parser.parse_args()
fan_in = 2
fan_out = 3
glorot_scale = 1. / max(1., (fan_in + fan_out) / 2.)
glorot_limit = limit = np.sqrt(3.0 * glorot_scale)
mode = args.mode
tf.set_random_seed(0)
if mode in [4,5]:
np.random.seed(0)
print(mode)
if mode == 0:
variable = tf.compat.v1.get_variable(name='variable', shape=[fan_in, fan_out])
elif mode == 1:
variable = tf.compat.v1.get_variable(name='variable', shape=[fan_in, fan_out],
initializer=tf.glorot_uniform_initializer(seed=0))
elif mode == 2:
variable = tf.Variable(tf.glorot_uniform_initializer(seed=0)(
shape=[fan_in, fan_out], dtype=tf.float32))
elif mode == 3:
variable = tf.random.stateless_uniform(
shape=(fan_in,fan_out), seed=[0, 0],
minval=-glorot_limit, maxval=glorot_limit)
elif mode == 4:
variable = tf.Variable(
np.random.uniform(-glorot_limit, glorot_limit, (fan_in, fan_out)),
dtype=tf.float32)
elif mode == 5:
variable = np.array(np.random.uniform(-glorot_limit, glorot_limit, (fan_in, fan_out)),
dtype=np.float32)
if mode in range(5):
sess = tf.compat.v1.Session()
sess.run(tf.compat.v1.global_variables_initializer())
variable = sess.run(variable)
print(variable)
The outputs are the following:
0
[[-0.39295787 0.4208691 0.53050697]
[ 0.8091326 1.007618 0.95607924]]
1
[[-1.0521597 -1.0800165 -0.6794561 ]
[ 0.60745895 -0.17927146 0.5341264 ]]
2
[[-1.0521597 -1.0800165 -0.6794561 ]
[ 0.60745895 -0.17927146 0.5341264 ]]
3
[[-0.6331334 0.45134532 0.657406 ]
[ 0.59327614 -0.24409002 0.81141686]]
4
[[ 0.10694503 0.4714563 0.22514328]
[ 0.09833413 -0.16726395 0.31963798]]
5
[[ 0.10694503 0.4714563 0.22514328]
[ 0.09833413 -0.16726395 0.31963798]]
答案1
得分: 0
原来底层的伪随机数生成器不同:TensorFlow 默认使用 Philox,而 numpy 默认使用 Mersenne-Twister。因此,在相同的随机种子下,它们会生成不同的均匀分布数值。
英文:
Turns out that the underlying pseudo random number generators differ: While TensorFlow uses Philox by default, numpy uses the Mersenne-Twister by default. So, under the same random seed they produce different values for a uniform distribution.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论