创建一个内容大于 2GB 的张量原型 [英] create a tensor proto whose content is larger than 2GB
问题描述
我创建了一个大小为 (2^22, 256) 的 ndarray (W),我尝试使用这个数组作为我的权重 matirx 初始化:
I created a ndarray (W) which size is (2^22, 256), and I tried to use this array as my initialization of weight matirx using:
w = tf.Variable(tf.convert_to_tensor(W))
然后,tensorflow 引发了一个错误:ValueError:无法创建内容大于 2GB 的张量原型.
then, the tensorflow raised a error: ValueError: Cannot create a tensor proto whose content is larger than 2GB.
我该如何解决这个问题?附注.我的权重矩阵必须使用 (2^22, 256) 矩阵进行初始化.谢谢:)
How can I fix this problem? PS. my weight matrix must using that (2^22, 256) matrix for initializing. THX :)
推荐答案
Protobuf 有 2GB 的硬限制.2^22*256 个浮点数为 4GB.您的问题是,您将通过
Protobuf has a hard limit of 2GB. And 2^22*256 floats are 4GB. Your problem is, that you are going to embed the initial value into the graph-proto by
import tensorflow as tf
import numpy as np
w_init = np.random.randn(2**22, 256).astype(np.float32)
w = tf.Variable(tf.convert_to_tensor(w_init))
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
print sess.run(tf.reduce_sum(w))
导致
ValueError: Cannot create a tensor proto whose content is larger than 2GB.
上面的这个图定义基本上是说:该图有一个占用 4GB 的变量,这里是确切的值:..."
This graph definition above is basically saying: "The graph has a variable occupying 4GB and here are the exact values: ..."
相反,你应该写
import tensorflow as tf
import numpy as np
w_init = np.random.randn(2**22, 256).astype(np.float32)
w_plhdr = tf.placeholder(dtype=tf.float32, shape=[2**22, 256])
w = tf.get_variable('w', [2**22, 256])
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
sess.run(w.assign(w_plhdr), {w_plhdr: w_init})
print sess.run(tf.reduce_sum(w))
这样,您的变量拥有 4GB 的值,但图形只有知识:嘿,有一个大小为 4 GB 的变量.只是不要关心图形定义中的确切值.因为有一个以后无论如何都要覆盖这些值的操作.".
This way, your variable holds 4GB of value but the graph only has the knowledge: "Hey, there is a variable of size 4 GB. Just don't care about the exact values within the graph definition. Because there is an operation to overwrite these values anyway later.".
这篇关于创建一个内容大于 2GB 的张量原型的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!