为什么在 TensorFlow 的内存中多次存储 `tf.constant()` 的值? [英] Why is the value of a `tf.constant()` stored multiple times in memory in TensorFlow?

问题描述

我阅读(在 TensorFlow 中):

<块引用>

的价值tf.constant() 在内存中多次存储.

为什么一个 tf.constant() 的值会在内存中多次存储?

解决方案

因为常量张量的数据被嵌入到图形定义中.这意味着这些数据既存储在维护图定义的客户端中，又存储在运行时中，后者为所有张量分配自己的内存.

IE，试试

a = tf.constant([1,2])tf.get_default_graph().as_graph_def()

你会看到

 数据类型:DT_INT32张量形状{昏暗{尺寸:2}}张量内容:\001\000\000\000\002\000\000\000"}

tensor_content 字段是原始内容，与 np.array([1,2], dtype=np.int32).tobytes() 相同.

现在，要查看运行时分配，您可以使用 export TF_CPP_MIN_LOG_LEVEL=1 运行.

如果你使用 a 来评估任何东西，你会看到类似这样的东西

2017-02-24 16:13:58: I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorOutput { step_id: 1 kernel_name: "Const_1/_1" 张量 { dtype: DT_INT32 shape {暗淡{大小:2}}分配描述{请求字节:8分配字节:256分配器名称:cuda_host_bfc"分配ID:1ptr:8605532160}}}

这意味着运行时要求分配 8 个字节，而 TF 实际分配了 256 个字节.(目前实际分配多少数据的选择有些随意 - bfc_allocator.cc )

在图中嵌入常量可以更轻松地进行一些基于图的优化，例如 不断折叠.但这也意味着大常数是低效的.此外，使用大常量是图形大小超过 2GB 限制的常见原因.

I read that (in TensorFlow):

the value of a tf.constant() is stored multiple times in memory.

Why is the value of a tf.constant() stored multiple times in memory?

解决方案

Because data for a constant tensor is embedded into graph definition. This means this data is stored both in the client, which maintains the graph definition, and in the runtime, which allocates it's own memory for all tensors.

IE, try

a = tf.constant([1,2])
tf.get_default_graph().as_graph_def()

You'll see

    dtype: DT_INT32
    tensor_shape {
      dim {
        size: 2
      }
    }
    tensor_content: "\001\000\000\000\002\000\000\000"
  }

The tensor_content field is the raw content, same as np.array([1,2], dtype=np.int32).tobytes().

Now, to see the runtime allocation, you can run with export TF_CPP_MIN_LOG_LEVEL=1.

If you evaluate anything using a you'll see something like this

2017-02-24 16:13:58: I tensorflow/core/framework/log_memory.cc:35] __LOG_MEMORY__ MemoryLogTensorOutput { step_id: 1 kernel_name: "Const_1/_1" tensor { dtype: DT_INT32 shape { dim { size: 2 } } allocation_description { requested_bytes: 8 allocated_bytes: 256 allocator_name: "cuda_host_bfc" allocation_id: 1 ptr: 8605532160 } } }

This means the runtime asked to allocate 8 bytes, and TF actually allocated 256 bytes. (the choices on how much data to actually allocate are somewhat arbitrary at the moment - bfc_allocator.cc )

Having constants embedded in the graph makes it easier to do some graph-based optimizations like constant folding . But this also means that large constants are inefficient. Also, using large constants is a common cause of exceeding 2GB limit for size of graph.

这篇关于为什么在 TensorFlow 的内存中多次存储 `tf.constant()` 的值?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！