PyTorch Autograd 自动微分功能 [英] PyTorch Autograd automatic differentiation feature

问题描述

我只是想知道，PyTorch 如何跟踪张量上的操作(在 .requires_grad 被设置为 True 之后，它后来如何自动计算梯度.请帮助我理解 autograd 背后的想法.谢谢.

I am just curious to know, how does PyTorch track operations on tensors (after the .requires_grad is set as True and how does it later calculate the gradients automatically. Please help me understand the idea behind autograd. Thanks.

推荐答案

这是个好问题！通常，自动微分(AutoDiff)的思想是基于多变量链式法则，即.
这意味着您可以通过代理"变量 y 表示 x 相对于 z 的导数；事实上，这允许您将几乎所有操作分解为一堆更简单(或原子)的操作，然后可以链接"在一起.
现在，像 Autograd 这样的 AutoDiff 包所做的只是存储这种原子操作块的导数，例如除法、乘法等.然后，在运行时，您提供的前向传递公式(由多个这些块组成)可以轻松转换为精确的导数.同样，如果您认为 AutoDiff 没有完全按照您的意愿行事，您也可以为您自己的操作提供衍生产品.

That's a great question! Generally, the idea of automatic differentiation (AutoDiff) is based on the multivariable chain rule, i.e. .
What this means is that you can express the derivative of x with respect to z via a "proxy" variable y; in fact, that allows you to break up almost any operation in a bunch of simpler (or atomic) operations that can then be "chained" together.
Now, what AutoDiff packages like Autograd do, is simply to store the derivative of such an atomic operation block, e.g., a division, multiplication, etc. Then, at runtime, your provided forward pass formula (consisting of multiple of these blocks) can be easily turned into an exact derivative. Likewise, you can also provide derivatives for your own operations, should you think AutoDiff does not exactly do what you want it to.

AutoDiff 相对于诸如有限差分之类的导数近似的优势在于，这是一个精确解决方案.

The advantage of AutoDiff over derivative approximations like finite differences is simply that this is an exact solution.

如果您对其内部的工作方式更感兴趣，我强烈推荐 AutoDidact 项目，它的目标是简化自动微分器的内部结构，因为通常还涉及很多代码优化.此外，这一组幻灯片来自我的讲座take对理解很有帮助.

If you are further interested in how it works internally, I highly recommend the AutoDidact project, which aims to simplify the internals of an automatic differentiator, since there is usually also a lot of code optimization involved. Also, this set of slides from a lecture I took was really helpful in understanding.

这篇关于PyTorch Autograd 自动微分功能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！