在Java中读取和处理大文本文件？ [英] Read and process big text file in Java?

问题描述

我想读一个非常大的文本文件（一个web应用程序的日志文件），并做一些处理。

有没有任何框架来帮助做这样的工作？

该文件是100M +，我将使用墙身线程？

问候

'DIV CLASS =h2_lin>解决方案

在你的情况多线程也不会有多大的问题是I / O密集型而非CPU绑定（当然，除非你想对内存中的文本文件进行大量处理，然后将其写回）。如果关心的是读取文件，通常100 MB是大型系统可以处理的。如果这是文件的大小，并且您正在Unix机器上运行，请查看您是否可以在64位虚拟机下运行代码。当然，这不是一个永久的解决方案。

一个可扩展的解决方案是让您逐行阅读文件，只保留您想要的数据，并最终处理只有数据（假设你可以做离线处理）。 Little Bobby Tables的方法是一个很好的方法，因为它给你一个持续的处理时间（实际上它将是O（n），其中n是要处理的行数）。

I want to read a very big text file(a log file of a web app)and do some processing.

Is there any Framework to help doing such work ?

The file is 100M+,shall I use mutil-thread ?

best regards

解决方案

In your case multi-threading will not help much as the problem is I/O bound rather than CPU bound (well, unless you are trying to do a lot of processing of the text file in memory and then write it back). If the concern is reading the file, generally 100 MB is something a large system can handle. If that is the size of the file and you are running on a Unix machine, see if you can run your code under the 64 bit VM. Of course this is not really a permanent solution.

A scalable solution is for you to read the file line by line and keep only the data that you want and finally work on that data alone (assuming you can do off-line processing). The approach by Little Bobby Tables is a good one since it gives you a constant processing time (actually it will be O(n) where n is the number of lines to process).

这篇关于在Java中读取和处理大文本文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！