在文本文件Java中写入巨大数据的最快方式 [英] Fastest way to write huge data in text file Java

问题描述

我必须在文本[csv]文件中写入大量的数据。我使用BufferedWriter写入数据，大约需要40秒才能写入174 MB的数据。这是Java可以提供的最快速度吗？

  bufferedWriter = new BufferedWriter（new FileWriter（fileName.csv））; 
  

注意：这40秒包括迭代和提取记录的时间从结果也。 :) 174 MB是结果集中的40万行。

解决方案

您可以尝试删除BufferedWriter，并直接使用FileWriter。在现代系统中，您很可能只是写入驱动器的缓存内存。

需要4-5秒的时间才能写入175MB 400万个字符串） - 这是一个双核2.4GHz戴尔运行Windows XP，具有80GB，7200-RPM Hitachi磁盘。

你可以隔离多少时间是记录检索和文件写入多少？

  import java.io.BufferedWriter; 
 import java.io.File; 
 import java.io.FileWriter; 
 import java.io.IOException; 
 import java.io.Writer; 
 import java.util.ArrayList; 
 import java.util.List; 
 
 public class FileWritingPerfTest {
 
 
 private static final int ITERATIONS = 5; 
 private static final double MEG =（Math.pow（1024,2））; 
 private static final int RECORD_COUNT = 4000000; 
 private static final String RECORD =帮助我被困在幸运饼干工厂\; 
 private static final int RECSIZE = RECORD.getBytes（）。length; 
 
 public static void main（String [] args）throws异常{
列表< String> records = new ArrayList< String>（RECORD_COUNT）; 
 int size = 0; （int i = 0; i< RECORD_COUNT; i ++）{
 records.add（RECORD）; 
 
 size + = RECSIZE; 
} 
 System.out.println（records.size（）+'records'）; 
 System.out.println（size / MEG +MB）; （int i = 0; i< ITERATIONS; i ++）{
 System.out.println（\\\
Iteration+ i）; 
 
 
 
 writeRaw（records）; 
 writeBuffered（records，8192）; 
 writeBuffered（records，（int）MEG）; 
 writeBuffered（records，4 *（int）MEG）; 
} 
} 
 
 private static void writeRaw（List< String> records）throws IOException {
文件文件= File.createTempFile（foo，.txt ）; 
 try {
 FileWriter writer = new FileWriter（file）; 
 System.out.print（Writing raw ...）; 
写（记录，作者）; 
} finally {
 //如果你以后要检查文件$ ​​b $ b file.delete（）; 
} 
} 
 
 private static void writeBuffered（List< String> records，int bufSize）throws IOException {
文件文件= File.createTempFile（foo 。文本）; 
 try {
 FileWriter writer = new FileWriter（file）; 
 BufferedWriter bufferedWriter = new BufferedWriter（writer，bufSize）; 
 
 System.out.print（Writing buffered（buffer size：+ bufSize +）...）; 
 write（records，bufferedWriter）; 
} finally {
 //如果你以后要检查文件$ ​​b $ b file.delete（）; 
} 
} 
 
 private static void write（List< String> records，Writer writer）throws IOException {
 long start = System.currentTimeMillis（）; 
 for（String record：records）{
 writer.write（record）; 
} 
 writer.flush（）; 
 writer.close（）; 
 long end = System.currentTimeMillis（）; 
 System.out.println（（end-start）/ 1000f +seconds）; 
} 
} 
  

I have to write huge data in text[csv] file. I used BufferedWriter to write the data and it took around 40 secs to write 174 mb of data. Is this the fastest speed java can offer?

bufferedWriter = new BufferedWriter ( new FileWriter ( "fileName.csv" ) );

Note: These 40 secs include the time of iterating and fetching the records from resultset as well. :) . 174 mb is for 400000 rows in resultset.

解决方案

You might try removing the BufferedWriter and just using the FileWriter directly. On a modern system there's a good chance you're just writing to the drive's cache memory anyway.

It takes me in the range of 4-5 seconds to write 175MB (4 million strings) -- this is on a dual-core 2.4GHz Dell running Windows XP with an 80GB, 7200-RPM Hitachi disk.

Can you isolate how much of the time is record retrieval and how much is file writing?

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.util.ArrayList;
import java.util.List;

public class FileWritingPerfTest {


private static final int ITERATIONS = 5;
private static final double MEG = (Math.pow(1024, 2));
private static final int RECORD_COUNT = 4000000;
private static final String RECORD = "Help I am trapped in a fortune cookie factory\n";
private static final int RECSIZE = RECORD.getBytes().length;

public static void main(String[] args) throws Exception {
    List<String> records = new ArrayList<String>(RECORD_COUNT);
    int size = 0;
    for (int i = 0; i < RECORD_COUNT; i++) {
        records.add(RECORD);
        size += RECSIZE;
    }
    System.out.println(records.size() + " 'records'");
    System.out.println(size / MEG + " MB");

    for (int i = 0; i < ITERATIONS; i++) {
        System.out.println("\nIteration " + i);

        writeRaw(records);
        writeBuffered(records, 8192);
        writeBuffered(records, (int) MEG);
        writeBuffered(records, 4 * (int) MEG);
    }
}

private static void writeRaw(List<String> records) throws IOException {
    File file = File.createTempFile("foo", ".txt");
    try {
        FileWriter writer = new FileWriter(file);
        System.out.print("Writing raw... ");
        write(records, writer);
    } finally {
        // comment this out if you want to inspect the files afterward
        file.delete();
    }
}

private static void writeBuffered(List<String> records, int bufSize) throws IOException {
    File file = File.createTempFile("foo", ".txt");
    try {
        FileWriter writer = new FileWriter(file);
        BufferedWriter bufferedWriter = new BufferedWriter(writer, bufSize);

        System.out.print("Writing buffered (buffer size: " + bufSize + ")... ");
        write(records, bufferedWriter);
    } finally {
        // comment this out if you want to inspect the files afterward
        file.delete();
    }
}

private static void write(List<String> records, Writer writer) throws IOException {
    long start = System.currentTimeMillis();
    for (String record: records) {
        writer.write(record);
    }
    writer.flush();
    writer.close();
    long end = System.currentTimeMillis();
    System.out.println((end - start) / 1000f + " seconds");
}
}

这篇关于在文本文件Java中写入巨大数据的最快方式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！