从二进制文件中读取大量 int 的最快方法 [英] Fastest way to read huge number of int from binary file

问题描述

我在嵌入式 Linux 设备上使用 Java 1.5，并且想要读取具有 2MB 整数值的二进制文件.(现在是 4bytes Big Endian，但我可以决定格式)

I use Java 1.5 on an embedded Linux device and want to read a binary file with 2MB of int values. (now 4bytes Big Endian, but I can decide, the format)

Using DataInputStream via BufferedInputStream using dis.readInt())，这 500 000 次调用需要 17s 读取，但文件读入一个大字节缓冲区需要 5 秒.

Using DataInputStream via BufferedInputStream using dis.readInt()), these 500 000 calls needs 17s to read, but the file read into one big byte buffer needs 5 seconds.

如何更快地将该文件读入一个巨大的 int[] 中?

读取过程不应额外使用超过 512 kb.

The reading process should not use more than additionally 512 kb.

以下使用 nio 的代码并不比 java io 中的 readInt() 方法快.

This code below using nio is not faster than the readInt() approach from java io.

    // asume I already know that there are now 500 000 int to read:
    int numInts = 500000;
    // here I want the result into
    int[] result = new int[numInts];
    int cnt = 0;

    RandomAccessFile aFile = new RandomAccessFile("filename", "r");
    FileChannel inChannel = aFile.getChannel();

    ByteBuffer buf = ByteBuffer.allocate(512 * 1024);

    int bytesRead = inChannel.read(buf); //read into buffer.

    while (bytesRead != -1) {

      buf.flip();  //make buffer ready for get()

      while(buf.hasRemaining() && cnt < numInts){
       // probably slow here since called 500 000 times
          result[cnt] = buf.getInt();
          cnt++;
      }

      buf.clear(); //make buffer ready for writing
      bytesRead = inChannel.read(buf);
    }


    aFile.close();
    inChannel.close();

更新:对答案的评估:

在 PC 上，使用 IntBuffer 方法的内存映射是我设置中最快的.
在没有 jit 的嵌入式设备上，java.io DataiInputStream.readInt() 快一些(17 秒，而带有 IntBuffer 的 MemMap 为 20 秒)

On PC the Memory Map with IntBuffer approach was the fastest in my set up.
On the embedded device, without jit, the java.io DataiInputStream.readInt() was a bit faster (17s, vs 20s for the MemMap with IntBuffer)

最终结论:通过算法更改更容易实现显着的加速.(init 的小文件)

Final Conclusion: Significant speed up is easier to achieve via Algorithmic change. (Smaller file for init)

推荐答案

我不知道这是否会比 Alexander 提供的更快，但您可以尝试映射文件.

I don't know if this will be any faster than what Alexander provided, but you could try mapping the file.

    try (FileInputStream stream = new FileInputStream(filename)) {
        FileChannel inChannel = stream.getChannel();

        ByteBuffer buffer = inChannel.map(FileChannel.MapMode.READ_ONLY, 0, inChannel.size());
        int[] result = new int[500000];

        buffer.order( ByteOrder.BIG_ENDIAN );
        IntBuffer intBuffer = buffer.asIntBuffer( );
        intBuffer.get(result);
    }

这篇关于从二进制文件中读取大量 int 的最快方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！