异步写入R中的文件 [英] Asynchronous writes to a file in R

问题描述

谁能给我提供关于如何异步;一个非常大的文件的组块，每个被具有相同功能在多个集群（8-500）处理写一些指导。该文件是大约25 GB，所以太大，无法在内存中。

Could anyone provide me with some guidance on how to asynchronous writes from chunks of a very large file, each of which is being processed with the same function in a number of clusters (8-500). The file is approx 25 GB, so too large to fit in memory.

我们目前正在使用的'雪（秋季）和mmap的包，但无法通过一些瓶颈得到的。同步写入做工精细，但异步写入会更有效。

We are currently using the 'snow(fall)' and 'mmap' packages, but aren't able to get through some bottlenecks. Synchronous writes work fine, but asynchronous writes would be much more efficient.

我不知道是什么的更多信息，你所需要的，但请随时问我！

I'm not sure what more information you need, but please feel free to ask me!

推荐答案

也许你可以使用

sink("filename.any_extension")

使其尽快从命令收到，并进一步利用这将输出结果中的文本文件来处理你的结果可以土色

so using this will output the result in the textfile as soon as it receives from command and further to process your result one can tuse

readLines("filename.any_extension",n=-1,warn=TRUE).

我希望这有助于。

I hope this helps.

这篇关于异步写入R中的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！