强制 R(和 Rstudio)使用 Windows 上的虚拟内存 [英] Forcing R (and Rstudio) to use the virtual memory on Windows

问题描述

我正在处理大型数据集，并且 R 经常会产生一个错误，告诉它无法分配该大小的向量或它没有足够的内存.

I'm working with large datasets and quite often R produces an error telling it can't allocate a vector of that size or it doesn't have enough memory.

我的计算机有 16GB 内存(Windows 10)，我正在处理大约 4GB 的数据集，但某些操作需要大量内存，例如将数据集从宽格式转换为长格式.在某些情况下，我可以使用 gc() 来释放一些内存，但很多时候这还不够.

My computer has 16GB RAM (Windows 10) and I'm working with datasets of around 4GB but some operations need a lot of memory, for example converting the dataset from wide format to long. In some situations I can use gc() to realease some memory but many times it's not enough.

有时我可以将数据集拆分成更小的块，但有时我需要一次处理所有表.

Sometimes I can break the dataset on smaller chunks but sometimes I need to work with all the table at once.

我听说 Linux 用户没有这个问题，但 Windows 呢?

I've read that Linux users don't have this problem, but what about Windows?

我尝试在 SSD (200GB) 上设置一个大页面文件，但我发现 R 根本不使用它.

I've tried setting a large pagefile on a SSD (200GB) but I've found that R doesn't use it at all.

我可以看到任务管理器，当内存消耗达到 16GB 时 R 停止工作.页面文件的大小似乎没有任何区别.

I can see the task manager and when the memory consumption reaches 16GB R stops working. The size of the pagefile doesn't seem to make any difference.

如何强制 R 使用页面文件?我需要用一些特殊的标志自己编译它吗?

How can I force R to use the pagefile? Do I need to compile it myself with some special flags?

PD:我的经验是删除一个对象 rm() 然后使用 gc() 不会恢复所有内存.当我对大型数据集执行操作时，无论我是否使用 gc()，我的计算机每一步的可用内存越来越少.

PD: My experience is that deleting an object rm() and later using gc() doesn't recover all the memory. As I perform operations with large datasets my computer has less and less free memory at every step, no matter if I use gc().

PD2:我希望不会听到诸如您需要更多 RAM 内存"之类的琐碎解决方案

PD2: I expect not to hear trivial solutions like "you need more RAM memory"

PD3:我一直在测试，问题只发生在 Rstudio 中.如果我直接使用 R 效果很好.有谁知道如何在 RStudio 中做到这一点.

PD3: I've been testing and the problem only happens in Rstudio. If I use directly R it works well. Does anybody know how to do it in RStudio.

推荐答案

为了让它在每次启动 RStudio 时自动工作，带有 R_MAX_MEM_SIZE 的解决方案将被忽略，无论是作为环境变量创建还是在 .Rprofile 中创建.

In order to get it working automatically every time you start RStudio the solution with R_MAX_MEM_SIZE is ignored, both if created as an environment variable or if created inside the .Rprofile.

写入 memory.limit(64000) 也被忽略.

Writing memory.limit(64000) is ignored too.

正确的方法是在文件 .Rprofile 中添加以下行

The proper way is adding the following line in the file .Rprofile

invisible(utils::memory.limit(64000))

或者任何你想要的数字.

or whatever number you want.

当然你需要有一个足够大的页面文件.该数字包括可用 RAM 和可用页面文件空间.

Of course you need to have a pagefile big enough. That number includes free RAM and free pagefile space.

使用页面文件速度较慢，但​​只会在需要时使用.

Using the pagefile is slower but it's going to be used only when needed.

我发现奇怪的是，它只能让您增加要使用的最大内存，但不允许您减少它.

Something strange I've found is that it only let's you increase the maximum memory to use but it doesn't allow you to decrease it.

这篇关于强制 R(和 Rstudio)使用 Windows 上的虚拟内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！