Pytorch:为什么`tensor`变量占用的内存这么小? [英] Pytorch: Why is the memory occupied by the `tensor` variable so small?

问题描述

在 Pytorch 1.0.0 中，我发现一个 tensor 变量占用的内存非常小.我想知道它是如何存储这么多数据的.这是代码.

In Pytorch 1.0.0, I found that a tensor variable occupies very small memory. I wonder how it stores so much data. Here's the code.

a = np.random.randn(1, 1, 128, 256)
b = torch.tensor(a, device=torch.device('cpu'))

a_size = sys.getsizeof(a)
b_size = sys.getsizeof(b)

a_size 是 262288.b_size 是 72.

a_size is 262288. b_size is 72.

推荐答案

答案分为两部分.来自 sys.getsizeof，首先

The answer is in two parts. From the documentation of sys.getsizeof, firstly

所有内置对象都将返回正确的结果，但对于第三方扩展，这不一定适用，因为它是特定于实现的.

All built-in objects will return correct results, but this does not have to hold true for third-party extensions as it is implementation specific.

所以对于张量来说，__sizeof__ 可能是未定义的或定义的与您预期的不同 - 此函数不是您可以依赖的.其次

so it could be that for tensors __sizeof__ is undefined or defined differently than you would expect - this function is not something you can rely on. Secondly

只考虑直接归因于对象的内存消耗，而不是它所引用的对象的内存消耗.

Only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to.

这意味着如果 torch.Tensor 对象仅持有对实际内存的引用，则不会在 sys.getsizeof 中显示.确实如此，如果您检查底层 storage 相反，你会看到预期的数字

which means that if the torch.Tensor object merely holds a reference to the actual memory, this won't show in sys.getsizeof. This is indeed the case, if you check the size of the underlying storage instead, you will see the expected number

import torch, sys
b = torch.randn(1, 1, 128, 256, dtype=torch.float64)
sys.getsizeof(b)
>> 72
sys.getsizeof(b.storage())
>> 262208

注意:我将 dtype 显式设置为 float64，因为这是 numpy 中的默认 dtype，而 torch 默认使用 float32.

Note: I am setting dtype to float64 explicitly, because that is the default dtype in numpy, whereas torch uses float32 by default.

这篇关于Pytorch:为什么`tensor`变量占用的内存这么小?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！