强制R(和Rstudio)在Windows上使用虚拟内存 [英] Forcing R (and Rstudio) to use the virtual memory on Windows

问题描述

我正在使用大型数据集，并且R经常会产生一个错误，告知它无法分配该大小的向量，或者它没有足够的内存.

I'm working with large datasets and quite often R produces an error telling it can't allocate a vector of that size or it doesn't have enough memory.

我的计算机具有16GB RAM(Windows 10)，我正在处理约4GB的数据集，但是某些操作需要大量内存，例如将数据集从宽格式转换为长格式. 在某些情况下，我可以使用gc()重新获得一些内存，但很多时候还不够.

My computer has 16GB RAM (Windows 10) and I'm working with datasets of around 4GB but some operations need a lot of memory, for example converting the dataset from wide format to long. In some situations I can use gc() to realease some memory but many times it's not enough.

有时我可以将数据集分成较小的块，但有时我需要一次处理所有表.

Sometimes I can break the dataset on smaller chunks but sometimes I need to work with all the table at once.

我已经读过Linux用户没有这个问题，但是Windows呢?

I've read that Linux users don't have this problem, but what about Windows?

我尝试在SSD(200GB)上设置一个大页面文件，但是我发现R根本不使用它.

I've tried setting a large pagefile on a SSD (200GB) but I've found that R doesn't use it at all.

我可以看到任务管理器，当内存消耗达到16GB时R停止工作.页面文件的大小似乎没有任何区别.

I can see the task manager and when the memory consumption reaches 16GB R stops working. The size of the pagefile doesn't seem to make any difference.

如何强制R使用页面文件? 我需要自己编写一些特殊标志吗?

How can I force R to use the pagefile? Do I need to compile it myself with some special flags?

PD:我的经验是删除对象rm()并随后使用gc()并不能恢复所有内存.当我对大型数据集执行操作时，无论是否使用gc()，计算机的每一步都有越来越少的可用内存.

PD: My experience is that deleting an object rm() and later using gc() doesn't recover all the memory. As I perform operations with large datasets my computer has less and less free memory at every step, no matter if I use gc().

PD2:我希望不会听到一些琐碎的解决方案，例如您需要更多的RAM内存"

PD2: I expect not to hear trivial solutions like "you need more RAM memory"

PD3:我一直在测试，该问题仅在Rstudio中发生.如果我直接使用R，效果很好.有人知道如何在RStudio中做到这一点.

PD3: I've been testing and the problem only happens in Rstudio. If I use directly R it works well. Does anybody know how to do it in RStudio.

推荐答案

为了使它在每次启动RStudio时自动运行，无论是作为环境变量创建还是在.Rprofile内部创建，都会忽略具有R_MAX_MEM_SIZE的解决方案.

In order to get it working automatically every time you start RStudio the solution with R_MAX_MEM_SIZE is ignored, both if created as an environment variable or if created inside the .Rprofile.

写内存.limit(64000)也被忽略.

Writing memory.limit(64000) is ignored too.

正确的方法是在文件.Rprofile中添加以下行

The proper way is adding the following line in the file .Rprofile

invisible(utils::memory.limit(64000))

或您想要的任何数字.

当然，您需要一个足够大的页面文件.该数字包括可用的RAM和可用的页面文件空间.

Of course you need to have a pagefile big enough. That number includes free RAM and free pagefile space.

使用页面文件的速度较慢，但​​仅在需要时才使用.

Using the pagefile is slower but it's going to be used only when needed.

我发现有些奇怪的是，它只能让您增加要使用的最大内存，但不允许您减少它.

Something strange I've found is that it only let's you increase the maximum memory to use but it doesn't allow you to decrease it.

这篇关于强制R(和Rstudio)在Windows上使用虚拟内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！