以最快的方式在CSV中写入100,000行 [英] Writing 100,000 lines in a csv in fastest way possible

问题描述

我正在尝试创建一个100 csv的zip文件。
我需要以最快的方式在单个csv中写入100,000行。
我正在使用openCSV，ZipEntry方法写入csv。

I m trying to create a zip of 100 csv. I need to write 100,000 lines in a single csv in fastest way possible. I am using openCSV, ZipEntry method to write to csv.

使用的某些代码：

ZipEntry zipentry = new ZipEntry(filename);
zos.putNextEntry(entry);
CSVWriter writer = new CsvWriter(new OutputStreamWriter(zos));
writer.writeNext(entries); //entries is single line of csv

当前写入单个行需要1.5秒csv和整个过程大约需要120-140秒才能创建完整的zip。

Currently its taking 1.5 secs to write a single csv and overall its taking around 120-140secs to create complete zip.

我已经调试了代码，并观察到代码中的其他计算并没有花费时间，但是写操作确实需要时间。

I have debugged the code, and observed that the other computations in the code is not taking time, but the write operation does takes time.

我尝试创建100,000行的列表，然后一次写入一个文件而不是直接流式传输。但是仍然需要相同的时间。

I have tried creating list of 100,000 lines and then writing one one file at a time instead if direct streaming. But still it takes same time.

请提出耗时较少的最快方法。 ;-（

Please suggest fastest method which takes less time. ;-(

推荐答案

BufferedOutputStream 大幅度提高了IO性能：

BufferedOutputStream improves IO performance by a large margin:

请参见

see Javadoc: BufferedOutputStream

通过设置这样的输出流，应用程序可以将字节写入底层输出

By setting up such an output stream, an application can write bytes to the underlying output stream without necessarily causing a call to the underlying system for each byte written.

OutputStream out = new BufferedOutputStream(zos, 1<<16);
CSVWriter writer = new CsvWriter(new OutputStreamWriter(out);

这篇关于以最快的方式在CSV中写入100,000行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！