ff软件包写入错误 [英] ff package write error

问题描述

我正在尝试使用R处理1909x139352数据集.由于我的计算机只有2GB RAM，因此对于常规方法而言，该数据集太大(500MB).因此，我决定使用ff包.但是，我遇到了一些麻烦.函数read.table.ffdf无法读取第一个数据块.它因下一个错误而崩溃:

I'm trying to work with a 1909x139352 dataset using R. Since my computer only has 2GB of RAM, the dataset turns out to be too big (500MB) for the conventional methods. So I decided to use the ff package. However, I've been having some troubles. The function read.table.ffdf is unable to read the first chunk of data. It crashes with the next error:

txtdata <- read.table.ffdf(file="/directory/myfile.csv", 
                           FUN="read.table", 
                           header=FALSE, 
                           sep=",", 
                          colClasses=c("factor",rep("integer",139351)), 
                          first.rows=100, next.rows=100, 
                          VERBOSE=TRUE)

  read.table.ffdf 1..100 (100)  csv-read=77.253sec
  Error en  ff(initdata = initdata, length = length, levels = levels, ordered = ordered,  : 
   write error

有人对发生的事情有任何了解吗?

Does anyone have any idea of what is going on?

推荐答案

此错误消息表明您有太多打开的文件.在ff中，ffdf中的每一列都是一个文件.您只能打开有限数量的文件-您已达到该数量.请参阅有关如何调试此FF错误的任何想法? .

This error message indicates that you have too many open files. In ff, every column in your ffdf is a file. You can only have a limited number of files open - and you have hit that number. See my reply on Any ideas on how to debug this FF error?.

因此，在您的情况下，仅使用read.table.ffdf将不起作用，因为您有139352列.但是可以将其导入ff，但是在RAM中获取数据时打开列时要小心，以免发生此问题.

So in your case, using simply read.table.ffdf won't work because you have 139352 columns. It is possible however to import it in ff but you need to be carefull when opening columns while getting data in RAM to avoid this issue.

这篇关于ff软件包写入错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！