从二进制文件中读取大量int的最快方法 [英] Fastest way to read huge number of int from binary file

问题描述

我在嵌入式Linux设备上使用Java 1.5，并希望读取具有2MB int值的二进制文件。 （现在4字节Big Endian，但我可以决定格式）

I use Java 1.5 on an embedded Linux device and want to read a binary file with 2MB of int values. (now 4bytes Big Endian, but I can decide, the format)

使用 DataInputStream 通过 BufferedInputStream 使用 dis.readInt（）），这500 000个调用需要读取17秒，但读入一个大字节缓冲区的文件需要5个秒。

Using DataInputStream via BufferedInputStream using dis.readInt()), these 500 000 calls needs 17s to read, but the file read into one big byte buffer needs 5 seconds.

如何将该文件更快地读入一个巨大的int []？

阅读过程不应超过额外512 kb。

The reading process should not use more than additionally 512 kb.

以下使用 nio 的代码不是比java io的readInt（）方法更快。

This code below using nio is not faster than the readInt() approach from java io.

    // asume I already know that there are now 500 000 int to read:
    int numInts = 500000;
    // here I want the result into
    int[] result = new int[numInts];
    int cnt = 0;

    RandomAccessFile aFile = new RandomAccessFile("filename", "r");
    FileChannel inChannel = aFile.getChannel();

    ByteBuffer buf = ByteBuffer.allocate(512 * 1024);

    int bytesRead = inChannel.read(buf); //read into buffer.

    while (bytesRead != -1) {

      buf.flip();  //make buffer ready for get()

      while(buf.hasRemaining() && cnt < numInts){
       // probably slow here since called 500 000 times
          result[cnt] = buf.getInt();
          cnt++;
      }

      buf.clear(); //make buffer ready for writing
      bytesRead = inChannel.read(buf);
    }


    aFile.close();
    inChannel.close();

更新：评估答案：

在PC上，使用IntBuffer方法的内存映射是我设置中最快的。

在嵌入式设备上，没有jit，java.io DataiInputStream.readInt（）是一个比较快（17秒，与使用IntBuffer的MemMap相比，20秒）

On PC the Memory Map with IntBuffer approach was the fastest in my set up.
On the embedded device, without jit, the java.io DataiInputStream.readInt() was a bit faster (17s, vs 20s for the MemMap with IntBuffer)

最终结论：
通过算法更改更容易实现显着的加速。 （初始化的文件较小）

Final Conclusion: Significant speed up is easier to achieve via Algorithmic change. (Smaller file for init)

推荐答案

我不知道这是否会比亚历山大提供的更快，但你可以尝试映射文件。

I don't know if this will be any faster than what Alexander provided, but you could try mapping the file.

    try (FileInputStream stream = new FileInputStream(filename)) {
        FileChannel inChannel = stream.getChannel();

        ByteBuffer buffer = inChannel.map(FileChannel.MapMode.READ_ONLY, 0, inChannel.size());
        int[] result = new int[500000];

        buffer.order( ByteOrder.BIG_ENDIAN );
        IntBuffer intBuffer = buffer.asIntBuffer( );
        intBuffer.get(result);
    }

这篇关于从二进制文件中读取大量int的最快方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！