最快的方式来增量读取大文件 [英] Fastest way to incrementally read a large file
问题描述
当给定一个MAX_BUFFER_SIZE的缓冲区和一个远远超过它的文件时，怎么能这样做：
- MAX_BUFFER_SIZE？
- 尽可能快地使用
我试过使用NIO
RandomAccessFile aFile = new RandomAccessFile（fileName，r）;
FileChannel inChannel = aFile.getChannel（）;
ByteBuffer buffer = ByteBuffer.allocate（CAPARICY）;
int bytesRead = inChannel.read（buffer）;
buffer.flip（）;
while（buffer.hasRemaining（））{
buffer.get（）;
}
buffer.clear（）;
bytesRead = inChannel.read（buffer）;
aFile.close（）;
和常规IO
InputStream in = new FileInputStream（fileName）;
long length = fileName.length（）;
if（length> Integer.MAX_VALUE）{
throw new IOException（File is too large！）;
}
byte [] bytes = new byte [（int）length];
int offset = 0;
int numRead = 0; （offset< bytes.length
&&（numRead = in.read（bytes，offset，bytes.length-offset））> = 0）{
while $ b offset + = numRead;
if（offset< bytes.length）{
throw new IOException（Could not complete read file+ fileName）;
}
in.close（）;
结果发现常规IO的执行速度比NIO 即可。我错过了什么？这是预期的吗？有没有更快的方式来读取缓冲区块中的文件？
最终，我正在处理一个大文件，我没有内存一次全部读取。相反，我想逐块读取它，然后用于处理。 解决方案
假设您需要把整个文件一次读入内存（就像你现在正在做的那样），既没有读小块也没有NIO能够帮助你。
事实上， d可能是最好的阅读大块 - 你的常规IO代码自动为你做。
你的NIO代码目前比较慢，因为你只读一个字节在一个时间（使用 buffer.get（）; ）。
如果你想以块，在流之间传输 - 这是一个没有NIO的标准方式：
InputStream is = ...;
OutputStream os = ...;
字节缓冲区[] =新字节[1024];
int读取; （（read = is.read（buffer））！= -1）{
os.write（buffer，0，read）;
while
这个缓冲区大小只有1 KB，但是可以传送无限量的数据。如果你扩展你的答案，详细说明你在功能层面上的实际操作，我可以进一步改进以获得更好的答案。 ）
When given a buffer of MAX_BUFFER_SIZE, and a file that far exceeds it, how can one:
- Read the file in blocks of MAX_BUFFER_SIZE?
- Do it as fast as possible
I tried using NIO
RandomAccessFile aFile = new RandomAccessFile(fileName, "r");
FileChannel inChannel = aFile.getChannel();
ByteBuffer buffer = ByteBuffer.allocate(CAPARICY);
int bytesRead = inChannel.read(buffer);
buffer.flip();
while (buffer.hasRemaining()) {
buffer.get();
}
buffer.clear();
bytesRead = inChannel.read(buffer);
aFile.close();
And regular IO
InputStream in = new FileInputStream(fileName);
long length = fileName.length();
if (length > Integer.MAX_VALUE) {
throw new IOException("File is too large!");
}
byte[] bytes = new byte[(int) length];
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead = in.read(bytes, offset, bytes.length - offset)) >= 0) {
offset += numRead;
}
if (offset < bytes.length) {
throw new IOException("Could not completely read file " + fileName);
}
in.close();
Turns out that regular IO is about 100 times faster in doing the same thing as NIO. Am i missing something? Is this expected? Is there a faster way to read the file in buffer chunks?
Ultimately i am working with a large file i don't have memory for to read it all at once. Instead, I'd like to read it incrementally in blocks that would then be used for processing.
解决方案 Assuming that you need to read the entire file into memory at once (as you're currently doing), neither reading smaller chunks nor NIO are going to help you here.
In fact, you'd probably be best reading larger chunks - which your regular IO code is automatically doing for you.
Your NIO code is currently slower, because you're only reading one byte at a time (using buffer.get();).
If you want to process in chunks - for example, transferring between streams - here is a standard way of doing it without NIO:
InputStream is = ...;
OutputStream os = ...;
byte buffer[] = new byte[1024];
int read;
while((read = is.read(buffer)) != -1){
os.write(buffer, 0, read);
}
This uses a buffer size of only 1 KB, but can transfer an unlimited amount of data.
(If you extend your answer with details of what you're actually looking to do at a functional level, I could further improve this to a better answer.)
这篇关于最快的方式来增量读取大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
