PyTorch Autograd自动区分功能 [英] PyTorch Autograd automatic differentiation feature

问题描述

我只是想知道，PyTorch如何跟踪张量上的操作(在.requires_grad设置为True之后，它以后如何自动计算梯度.请帮助我理解autograd背后的想法.谢谢.

I am just curious to know, how does PyTorch track operations on tensors (after the .requires_grad is set as True and how does it later calculate the gradients automatically. Please help me understand the idea behind autograd. Thanks.

推荐答案

这是一个很好的问题！ 通常，自动微分(AutoDiff)的想法是基于多变量链规则，即 .
这意味着您可以通过代理"变量y表示x相对于z的导数；实际上，这使您可以分解一堆更简单(或原子)的操作中的几乎所有操作，然后将这些操作链接"在一起.
现在，像Autograd这样的AutoDiff包所做的只是简单地存储这种原子运算块的导数，例如除法，乘法等. 然后，在运行时，可以轻松地将您提供的前向传递公式(由这些模块中的多个模块组成)转换为精确的导数.同样，如果您认为AutoDiff不能完全按照自己的意愿进行操作，则还可以为自己的操作提供衍生工具.

That's a great question! Generally, the idea of automatic differentiation (AutoDiff) is based on the multivariable chain rule, i.e. .
What this means is that you can express the derivative of x with respect to z via a "proxy" variable y; in fact, that allows you to break up almost any operation in a bunch of simpler (or atomic) operations that can then be "chained" together.
Now, what AutoDiff packages like Autograd do, is simply to store the derivative of such an atomic operation block, e.g., a division, multiplication, etc. Then, at runtime, your provided forward pass formula (consisting of multiple of these blocks) can be easily turned into an exact derivative. Likewise, you can also provide derivatives for your own operations, should you think AutoDiff does not exactly do what you want it to.

AutoDiff优于诸如有限差异之类的导数近似值的优点只是因为这是一个精确的解决方案.

The advantage of AutoDiff over derivative approximations like finite differences is simply that this is an exact solution.

如果您对它在内部的工作方式进一步感兴趣，我强烈建议您使用 AutoDidact项目，该项目旨在简化自动微分器的内部，因为通常还涉及很多代码优化. 另外，这是我在第一讲课中的一组幻灯片确实有助于理解.

If you are further interested in how it works internally, I highly recommend the AutoDidact project, which aims to simplify the internals of an automatic differentiator, since there is usually also a lot of code optimization involved. Also, this set of slides from a lecture I took was really helpful in understanding.

这篇关于PyTorch Autograd自动区分功能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！