如何使用numpy复制tf.get_variable的默认行为? (TensorFlow v1.15.0)

huangapple go评论64阅读模式
英文:

How to reproduce the default behaviour of tf.get_variable with numpy? (TensorFlow v1.15.0)

问题

使用tf.get_variable初始化的变量默认情况下是从Glorot/Xavier分布中采样的。

如何使用numpy.random复现该分布?

这可能归结为设置正确的随机种子 - 但是,我尝试了一下(请参见下面的最小工作示例),但没有成功。有人知道如何使下面示例中的模式输出相同,或者理解它们为什么不同吗?

import tensorflow as tf
import numpy as np
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--mode', type=int)
args = parser.parse_args()

fan_in = 2
fan_out = 3

glorot_scale = 1. / max(1., (fan_in + fan_out) / 2.)
glorot_limit = limit = np.sqrt(3.0 * glorot_scale)

mode = args.mode

tf.set_random_seed(0)

if mode in [4,5]:
    np.random.seed(0)
    
print(mode)
if mode == 0:
    variable = tf.compat.v1.get_variable(name='variable', shape=[fan_in, fan_out])
elif mode == 1:
    variable = tf.compat.v1.get_variable(name='variable', shape=[fan_in, fan_out],
                                         initializer=tf.glorot_uniform_initializer(seed=0))
elif mode == 2:
    variable = tf.Variable(tf.glorot_uniform_initializer(seed=0)(
                               shape=[fan_in, fan_out], dtype=tf.float32))
elif mode == 3:
    variable = tf.random.stateless_uniform(
                   shape=(fan_in,fan_out), seed=[0, 0],
                   minval=-glorot_limit, maxval=glorot_limit)
elif mode == 4:
    variable = tf.Variable(
                 np.random.uniform(-glorot_limit, glorot_limit, (fan_in, fan_out)),
                 dtype=tf.float32)
elif mode == 5:
    variable = np.array(np.random.uniform(-glorot_limit, glorot_limit, (fan_in, fan_out)), 
                                          dtype=np.float32)

if mode in range(5):
    sess = tf.compat.v1.Session()
    sess.run(tf.compat.v1.global_variables_initializer())
    variable = sess.run(variable)
print(variable)

输出如下:

0
[[-0.39295787  0.4208691   0.53050697]
 [ 0.8091326   1.007618    0.95607924]]
1
[[-1.0521597  -1.0800165  -0.6794561 ]
 [ 0.60745895 -0.17927146  0.5341264 ]]
2
[[-1.0521597  -1.0800165  -0.6794561 ]
 [ 0.60745895 -0.17927146  0.5341264 ]]
3
[[-0.6331334   0.45134532  0.657406  ]
 [ 0.59327614 -0.24409002  0.81141686]]
4
[[ 0.10694503  0.4714563   0.22514328]
 [ 0.09833413 -0.16726395  0.31963798]]
5
[[ 0.10694503  0.4714563   0.22514328]
 [ 0.09833413 -0.16726395  0.31963798]]
英文:

Variables initialized with tf.get_variable are (to my knowledge), per default, sampled from a Glorot/Xavier distribution.

How can one reproduce that distribution with numpy.random?

It probably boils down to setting the correct random seeds - which, however, I do not succeed in (see the following minimal working example). Does someone know how to make the outputs of the modes in the example below the same or understand why they differ?

import tensorflow as tf
import numpy as np
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--mode', type=int)
args = parser.parse_args()

fan_in = 2
fan_out = 3

glorot_scale = 1. / max(1., (fan_in + fan_out) / 2.)
glorot_limit = limit = np.sqrt(3.0 * glorot_scale)

mode = args.mode

tf.set_random_seed(0)

if mode in [4,5]:
    np.random.seed(0)
    
print(mode)
if mode == 0:
    variable = tf.compat.v1.get_variable(name='variable', shape=[fan_in, fan_out])
elif mode == 1:
    variable = tf.compat.v1.get_variable(name='variable', shape=[fan_in, fan_out],
                                         initializer=tf.glorot_uniform_initializer(seed=0))
elif mode == 2:
    variable = tf.Variable(tf.glorot_uniform_initializer(seed=0)(
                               shape=[fan_in, fan_out], dtype=tf.float32))
elif mode == 3:
    variable = tf.random.stateless_uniform(
                   shape=(fan_in,fan_out), seed=[0, 0],
                   minval=-glorot_limit, maxval=glorot_limit)
elif mode == 4:
    variable = tf.Variable(
                 np.random.uniform(-glorot_limit, glorot_limit, (fan_in, fan_out)),
                 dtype=tf.float32)
elif mode == 5:
    variable = np.array(np.random.uniform(-glorot_limit, glorot_limit, (fan_in, fan_out)), 
                                          dtype=np.float32)

if mode in range(5):
    sess = tf.compat.v1.Session()
    sess.run(tf.compat.v1.global_variables_initializer())
    variable = sess.run(variable)
print(variable)

The outputs are the following:

0
[[-0.39295787  0.4208691   0.53050697]
 [ 0.8091326   1.007618    0.95607924]]
1
[[-1.0521597  -1.0800165  -0.6794561 ]
 [ 0.60745895 -0.17927146  0.5341264 ]]
2
[[-1.0521597  -1.0800165  -0.6794561 ]
 [ 0.60745895 -0.17927146  0.5341264 ]]
3
[[-0.6331334   0.45134532  0.657406  ]
 [ 0.59327614 -0.24409002  0.81141686]]
4
[[ 0.10694503  0.4714563   0.22514328]
 [ 0.09833413 -0.16726395  0.31963798]]
5
[[ 0.10694503  0.4714563   0.22514328]
 [ 0.09833413 -0.16726395  0.31963798]]

答案1

得分: 0

原来底层的伪随机数生成器不同:TensorFlow 默认使用 Philox,而 numpy 默认使用 Mersenne-Twister。因此,在相同的随机种子下,它们会生成不同的均匀分布数值。

英文:

Turns out that the underlying pseudo random number generators differ: While TensorFlow uses Philox by default, numpy uses the Mersenne-Twister by default. So, under the same random seed they produce different values for a uniform distribution.

huangapple
  • 本文由 发表于 2023年6月12日 18:42:45
  • 转载请务必保留本文链接:https://go.coder-hub.com/76455856.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定