用缓冲区读取二进制文件 [英] Read binary file with a buffer

问题描述

我正在尝试读取包含100.000个不同对象的二进制文件.
使用 BufferedReader 缓冲具有相同内容的简单文本文件只需要2MB.

I'm trying to read a binary file containing 100.000 different objects.
buffering a simple text file with the same content takes only 2MB with a BufferedReader.

但是读取二进制文件最多需要700 MB，如果我增加读取对象的数量，则会收到 OutOfMemory 错误.

But reading the binary files takes up to 700 MB and i get OutOfMemory error if I increase the number of objects to read.

那么，如何在不使内存饱和的情况下读取文件并逐个获取对象呢?

So how to read the file and get the objects one by one without saturating the memory?

这是我正在测试的代码:

Here is the code I'm testing:

public static void main(String[] args) throws Exception {
    int i = 0;
    String path = "data/file.bin";
    InputStream file = new FileInputStream(path);
    InputStream buffer = new BufferedInputStream(file);
    ObjectInputStream in = new ObjectInputStream(buffer);
    Object obj = null;
    while( ( obj = in.readObject() ) != null && i < 100000 ){
        String str =  obj.toString();
        System.out.println( str );
        i++;
    }

    timeTkken();
}

// Function to get the amount of time/memory used by the script
private static final long startTime = System.currentTimeMillis();
private static final long MEGABYTE = 1024L * 1024L;
public static void timeTkken(){
    Runtime runtime = Runtime.getRuntime();
    long endTime = System.currentTimeMillis();
    long memory = runtime.totalMemory() - runtime.freeMemory();
    long megabytes = memory / MEGABYTE;
    System.out.println("It took " + megabytes + "mb in " + ( (endTime - startTime) /1000 ) + "s ("+ memory + (" bytes in ") + (endTime - startTime) + " ms)");

}

推荐答案

据我所知，ObjectInputStream将所有对象保留在缓存中，直到关闭流为止.因此，如果您的二进制文件约为207 MB，那么Java堆中的实际对象可能会轻易占用数GB的RAM，并且无法对其进行垃圾回收.问题出现在这里:是否需要将所有数据同时保存在RAM中?

As far as I know, ObjectInputStream keeps all the objects in cache until the stream is closed. So if your binary file is ~207 MB, then real objects in java heap may easily take several GBs of RAM and they can't be garbage collected. Here the question appears: Do you need all of your data to be held in RAM simultaneously?

如果否(您想读取一个对象，以某种方式处理它，将其丢弃并移至下一个对象)，我建议使用DataInputStream而不是ObjectInputStream.我不知道这种方法是否适用于您的情况，因为我不知道您的数据结构.如果您的数据是具有相同结构的记录的集合，则可以执行以下操作:

If no (you want to read an object, process it somehow, discard it and move to the next object), I would suggest using DataInputStream instead of ObjectInputStream. I don't know if this approach is applicable in your case since I don't know the structure of your data. If your data is a collection of records of the same structure, you may do the following:

    public class MyObject {
        private int age;
        private String name;

        public MyObject(int age, String name) {
            this.age = age;
            this.name = name;
        }
    }

    DataInputStream in = new DataInputStream(new BufferedInputStream(new FileInputStream("path.to.file")));
    // suppose that we store the total number of objects in the first 4 bytes of file
    int nObjects = in.readInt();
    for (int i = 0; i < nObjects; i++) {
        MyObject obj = new MyObject(in.readInt(), in.readUTF());
        // do some stuff with obj
    }

这篇关于用缓冲区读取二进制文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！