PyTorch中的重现性和性能 [英] Reproducibility and performance in PyTorch

问题描述

文档指出：

确定性模式可能会对性能产生影响，具体取决于您的模型。

我的问题就是说这里的表现是什么处理速度或模型质量（即最小损失）？换句话说，当设置手动种子并以确定性方式执行模型时，这会导致更长的训练时间，直到找到最小的损失，还是最小损失会比模型不确定时更糟？

出于完整性考虑，我通过设置以下所有属性来手动确定模型：

  def set_seed（seed）：
 torch.manual_seed（seed）
 torch.cuda.manual_seed_all（seed）
 torch.backends.cudnn.deterministic =真
 torch.backends .cudnn.benchmark = False 
 np.random.seed（seed）
 random.seed（seed）
 os.environ ['PYTHONHASHSEED'] = str（seed）
  

解决方案

性能是指运行时间； CuDNN有几种实现方式，当 cudnn.deterministic 设置为true时，您就是在告诉CuDNN您只需要确定性实现（或我们认为的实现）。简而言之，执行此操作时，在输入相同输入时，应该在CPU或相同系统上获得相同的结果 em>。为什么会影响性能？ CuDNN使用启发式方法来选择实现。因此，实际上取决于您的模型，CuDNN的行为如何。选择确定性的方法可能会影响运行时，因为可以说，在同一运行时选择它们的方式更快。<​​/ p>

---

关于您的代码段，我做了准确的播种，对于100多次DL实验，它一直工作良好（就可重复性而言）。

The documentation states:

Deterministic mode can have a performance impact, depending on your model.

My question is, what is meant by performance here. Processing speed or model quality (i.e. minimal loss)? In other words, when setting manual seeds and making the model perform in a deterministic way, does that cause longer training time until minimal loss is found, or is that minimal loss worse than when the model is non-deterministic?

For completeness' sake, I manually make the model deterministic by setting all of these properties:

def set_seed(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed(seed)
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)

解决方案

Performance refers to the run time; CuDNN has several ways of implementations, when cudnn.deterministic is set to true, you're telling CuDNN that you only need the deterministic implementations (or what we believe they are). In a nutshell, when you are doing this, you should expect the same results on the CPU or the GPU on the same system when feeding the same inputs. Why would it affect the performance? CuDNN uses heuristics for the choice of the implementation. So, it actually depends on your model how CuDNN will behave; choosing it to be deterministic may affect the runtime because their could have been, let's say, faster way of choosing them at the same point of running.

---

Concerning your snippet, I do the exact seeding, it has been working good (in terms of reproducibility) for 100+ DL experiments.

这篇关于PyTorch中的重现性和性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！