使用 fread() 导入大型 CSV (8 GB) 时“在字符串中嵌入 nul" [英] 'Embedded nul in string' when importing large CSV (8 GB) with fread()

问题描述

我有一个很大的 CSV 文件 (8.1 GB)，我正试图将其转换为 R.我使用 Python 的 csvkit in2csv 创建了 CSV，从 .txt 文件转换而来，但不知何故转换导致文件中出现空字符.我现在在导入时收到此错误:

I have a large CSV file (8.1 GB) that I'm trying to wrangle into R. I created the CSV using Python's csvkit in2csv, converted from a .txt file, but somehow the conversion led to null characters showing up in the file. I'm now getting this error when importing:

fread("file.csv", nrows = 100) 中的错误:在字符串中嵌入 nul:'ÿþrecd_zipc'

我我能够使用 read.csv 导入小块就好了，但那是因为它允许通过 fileEncoding 参数.

I am able to import small chunks just fine with read.csv though, but that's because it allows for UTF-16 encoding via the fileEncoding argument.

test <- read.csv("file.csv", nrows=100, fileEncoding="UTF-16LE")

不过，我不敢尝试使用 read.csv 导入 8 GB 文件.

I don't dare try to import an 8 GB file with read.csv, though.

于是我尝试了提供的解决方案 此处，在其中使用 sed s/\0//g file.csv >file2.csv 拉出空值.该命令执行得很好并填充了一个新的 8GB CSV 文件，但我收到了一个几乎相同的错误:

So I then tried the solution offered here, in which you use sed s/\0//g file.csv > file2.csv to pull the nulls out. The command performed just fine and populated a new 8GB CSV file, but I received a nearly-identical error:

fread("file2.csv", nrows = 100) 中的错误:在字符串中嵌入 nul: 'ÿþrecd_zipc,post_zi

所以，那没有用.我在这一点上难住了.考虑到文件的大小，我不能在整个过程中使用 read.csv，而且我不确定如何摆脱原始 CSV 中的空值.我什至不确定文件是如何编码为 UTF-16 的.在这一点上，任何建议或建议将不胜感激.

So, that didn't work. I'm stumped at this point. Considering the size of the file, I can't use read.csv on the whole thing, and I'm not sure how to get rid of the nulls in the original CSV. I'm not even sure how the file got encoded as UTF-16. Any suggestions or advice would be greatly appreciated at this point.

我使用的是 Windows 机器.

I'm on a Windows machine.

推荐答案

如果你使用的是 linux/mac，试试这个

If you're on linux/mac, try this

file <- "file.csv"
tt <- tempfile()  # or tempfile(tmpdir="/dev/shm")
system(paste0("tr < ", file, " -d '\000' >", tt))
fread(tt)

这篇关于使用 fread() 导入大型 CSV (8 GB) 时“在字符串中嵌入 nul"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！