TensorFlow Variable 和 TensorFlow Tensor 的实现区别 [英] Implementation difference between TensorFlow Variable and TensorFlow Tensor

问题描述

首先，我知道有人问了一个相关的问题

First of all, I am aware that a related question has been asked here.

然而，这个问题是关于实现和内部的.我正在阅读论文TensorFlow 之旅".以下两点引自那里:

However, this question is about the implementation and internals. I was reading the paper "A Tour of TensorFlow". The following two points are quoted from there:

1.

张量本身并不在内存中保存或存储值，但提供仅用于检索张量引用的值的接口.

A tensor itself does not hold or store values in memory, but provides only an interface for retrieving the value referenced by the tensor.

这向我表明，张量是一个对象，它只是存储指向操作结果的指针，并且在检索张量的结果或值时，它只是取消对该指针的引用.

This suggests to me that a Tensor is an object that simply stores the pointer to a result of an operation and, on retrieving the result or value of the tensor, it simply dereferences that pointer.

2.

变量可以被描述为存储张量的内存缓冲区的持久的、可变的句柄.因此，变量具有一定的形状和固定的类型.

Variables can be described as persistent, mutable handles to in-memory buffers storing tensors. As such, variables are characterized by a certain shape and a fixed type.

在这里我感到困惑，因为我认为，基于前一点，张量只是存储一个指针.如果它们只是指针，它们也可以是可变的.

At this I get confused because I thought, based on the previous point, that Tensors simply store a pointer. If they were simply pointers, they could be mutable as well.

准确地说，这些是我的问题:

To be precise these are my questions:

1. 内存缓冲区"是什么意思?
2. 句柄"是什么意思?
3. 我对张量内部的初步假设是否正确?
4. 张量和变量之间的基本内部实现区别是什么?为什么它们的声明方式不同，为什么这种差异对 TensorFlow 至关重要?

推荐答案

在解释张量和变量之间的区别之前，我们应该准确地了解张量"这个词在 TensorFlow 上下文中的含义:

Before explaining the distinction between tensors and variables, we should be precise about what the word "tensor" means in the context of TensorFlow:

• 在 Python API 中，tf.Tensor 对象表示 TensorFlow 操作的符号结果.例如，在表达式t = tf.matmul(x, y)中，t是一个tf.Tensor对象，表示乘法的结果x 和 y(它们本身可能是其他操作的符号结果，具体值，例如 NumPy 数组或变量).

• In the Python API, a tf.Tensor object represents the symbolic result of a TensorFlow operation. For example, in the expression t = tf.matmul(x, y), t is a tf.Tensor object representing the result of multiplying x and y (which may themselves be symbolic results of other operations, concrete values such as NumPy arrays, or variables).

在这种情况下，符号结果"比指向操作结果的指针更复杂.它更类似于一个函数对象，在调用时(即传递给 tf.Session.run())将运行必要的计算以产生该操作的结果，并将其返回给您一个具体的值(例如一个 NumPy 数组).

In this context, a "symbolic result" is more complicated than a pointer to the result of an operation. It is more analogous to a function object that, when called (i.e. passed to tf.Session.run()) will run the necessary computation to produce the result of that operation, and return it to you as a concrete value (e.g. a NumPy array).

在 C++ API 中，tensorflow::Tensor 对象表示多维数组的具体值.例如，MatMul 内核将两个二维 tensorflow::Tensor 对象作为输入，并生成一个二维 tensorflow::Tensor> 对象作为其输出.

In the C++ API, a tensorflow::Tensor object represents the concrete value of a multi-dimensional array. For example, the MatMul kernel takes two two-dimensional tensorflow::Tensor objects as inputs, and produces a single two-dimensional tensorflow::Tensor object as its output.

这种区别有点令人困惑，如果我们重新开始，我们可能会选择不同的名称(在其他语言 API 中，我们更喜欢名称 Output 作为符号结果和 Tensor> 获取具体值).

This distinction is a little confusing, and we might choose different names if we started over (in other language APIs, we prefer the name Output for a symbolic result and Tensor for a concrete value).

变量也存在类似的区别.在 Python API 中，tf.Variable 是变量的符号表示，它具有创建操作的方法，这些操作读取变量的当前值，并为其赋值.在 C++ 实现中，一个 对象是一个共享的、可变的 tensorflow::Tensor 对象的包装器.

A similar distinction exists for variables. In the Python API, a tf.Variable is the symbolic representation of a variable, which has methods for creating operations that read the current value of the variable, and assign values to it. In the C++ implementation, a tensorflow::Var object is a wrapper around a shared, mutable tensorflow::Tensor object.

了解这些背景后，我们可以解决您的具体问题:

With that context out the way, we can address your specific questions:

1. 内存缓冲区"是什么意思?

内存缓冲区只是已使用 TensorFlow 分配器分配的连续内存区域.tensorflow::Tensor 对象包含一个指向内存缓冲区的指针，该缓冲区保存该张量的值.缓冲区可以位于主机内存(即可从 CPU 访问)或设备内存(例如仅可从 GPU 访问)中，并且 TensorFlow 具有在这些内存空间之间移动数据的操作.

An in-memory buffer is simply a contiguous region of memory that has been allocated with a TensorFlow allocator. tensorflow::Tensor objects contain a pointer to an in-memory buffer, which holds the values of that tensor. The buffer could be in host memory (i.e. accessible from the CPU) or device memory (e.g. accessible only from a GPU), and TensorFlow has operations to move data between these memory spaces.

句柄"是什么意思?

在论文的解释中，handle"这个词用于几个不同的方式，这与 TensorFlow 使用该术语的方式略有不同.本文使用符号句柄"来指代tf.Tensor 对象，使用持久、可变句柄"来指代tf.Variable 对象.TensorFlow 代码库使用句柄"来指代有状态对象的名称(如 tf.FIFOQueue 或 tf.TensorArray)，无需复制所有值(即call-by-reference).

In the explanation in the paper, the word "handle" is used in a couple of different ways, which are slightly different from how TensorFlow uses the term. The paper uses "symbolic handle" to refer to a tf.Tensor object, and "persistent, mutable handle" to refer to a tf.Variable object. The TensorFlow codebase uses "handle" to refer to a name for a stateful object (like a tf.FIFOQueue or tf.TensorArray) that can be passed around without copying all of the values (i.e. call-by-reference).

我对张量内部的最初假设是否正确?

您的假设最符合 (C++) tensorflow::Tensor 对象的定义.(Python) tf.Tensor 对象更复杂，因为它指的是计算值的函数，而不是值本身.

Your assumption most closely matches the definition of a (C++) tensorflow::Tensor object. The (Python) tf.Tensor object is more complicated because it refers to a function for computing a value, rather than the value itself.

张量和变量之间的基本内部实现区别是什么?

在 C++ 中，tensorflow::Tensor 和 tensorflow::Var 非常相似；唯一不同的是 tensorflow::Var 还有一个 mutex 可以用来在更新时锁定变量.

In C++, a tensorflow::Tensor and tensorflow::Var are very similar; the only different is that tensorflow::Var also has a mutex that can be used to lock the variable when it is being updated.

在 Python 中，本质区别在于 tf.Tensor 被实现为数据流图，并且它是只读的(即通过调用 tf.Session.run()).tf.Variable 既可以读取(即通过评估其读取操作)也可以写入(例如通过运行分配操作).

In Python, the essential difference is that a tf.Tensor is implemented as a dataflow graph, and it is read-only (i.e. by calling tf.Session.run()). A tf.Variable can be both read (i.e. by evaluating its read operation) and written (e.g. by running an assign operation).

为什么它们的声明不同，为什么这种区别对 TensorFlow 至关重要?

张量和变量有不同的用途.张量(tf.Tensor 对象)可以表示数学表达式的复杂组合，例如神经网络中的损失函数或符号梯度.变量表示随时间更新的状态，例如训练期间的权重矩阵和卷积滤波器.虽然原则上您可以在没有变量的情况下表示模型的演化状态，但最终会得到一个非常大(且重复)的数学表达式，因此变量提供了一种实现模型状态的便捷方法，并且—例如—share与其他机器进行并行训练.

Tensors and variables serve different purposes. Tensors (tf.Tensor objects) can represent complex compositions of mathematical expressions, like loss functions in a neural network, or symbolic gradients. Variables represent state that is updated over time, like weight matrices and convolutional filters during training. While in principle you could represent the evolving state of a model without variables, you would end up with a very large (and repetetive) mathematical expression, so variables provide a convenient way to materialize the state of the model, and—for example—share it with other machines for parallel training.

这篇关于TensorFlow Variable 和 TensorFlow Tensor 的实现区别的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！