用Java增量读取日志文件 [英] Reading log files incrementally in Java

问题描述

我需要以增量方式读取和处理日志文件.关于如何在Java中执行此操作的任何建议?

我需要考虑所有可能的情况，例如文件翻转，不同的日志记录格式等.

解决方案

虽然已经很晚了，但是只是想写出我用来实现此功能的方法.

比方说，我们每隔5分钟就会开始定期读取文件的工作.

1. 第一次运行时，读取整个文件

2. 存储行数和文件的最后修改时间

   随后的作业运行变得很有趣.

3. 在下一次作业运行期间，检查文件是否被修改(使用文件的上次修改时间以及在较早的作业运行期间存储的文件).如果文件未修改，则什么也不做.

4. 如果文件被修改，我们只需要读取新行即可.我们具有先前工作的行数，因此可以使用它来确定要跳过的行数.

   到目前为止，如果将文件翻转了怎么办?

5. 假定文件具有翻转模式时的文件命名模式...

6. 获取所有与模式匹配的文件，并根据文件的最后修改时间以升序对其进行排序

7. 遍历文件，并从上次修改时间大于上次作业运行所存储时间的文件开始.聪明地使用存储的行数来跳过已经读取的行

8. 此后从新文件开始重置行数

就是这样！

对于某些奇怪的情况，您可能需要将IF条件放在几个地方.一种这样的情况是，当您遍历文件时，如果文件的上次修改时间与存储的时间完全相同，只需重置行数-以便从下一个/新文件的第一行开始.

后续作业运行的示例代码:

  for(文件文件:文件){< BR>if(file.lastModified()> storedLastModifiedTime){< BR>//您有要处理的文件，请注意行数< BR>} else if(file.lastModified()== storedLastModifiedTime){< BR>//重置存储的行数< BR>}< BR>}< BR> 

I have a requirement to read and process log file incrementally. Any suggestions on how to do this in Java?

I need to consider all possible scenarios like file rollover, different logging formats, etc.

解决方案

Though it's pretty late but just thought of writing the approach that I used to achieve this functionality.

Let's say we start a job to read a file periodically, after every 5 min.

1. During first run, read the entire file

2. Store line count and the last modified time of the file

   It becomes interesting for subsequent job runs.

3. During next job run, check if the file is modified (using file last modified time and the one stored during earlier job run). If the file is not modified, do nothing.

4. If the file is modified, we just need to read the new lines. We have the line count from the earlier job so use it to determine the number of lines to skip.

   So far so good, what if the file is rolled over?

5. Assuming we have the pattern for file naming when the file is rolled over...

6. Get all files matching the pattern and sort them in ascending order based on file last modified time

7. Iterate through the files and start with the one whose last modified time is greater than the time stored from the previous job run. Use stored line count smartly to skip the already read lines

8. Reset line count when you start with a new file thereafter

That's it!

You may need to put IF conditions at few places for some odd scenarios. One such scenario is when you are iterating through the files and if the file last modified time is exactly the same as the stored one, just reset the line count - so that it starts with the first line from the next/new file.

Sample code for subsequent job runs:

for(File file : files) {<BR>
  if(file.lastModified() > storedLastModifiedTime) {<BR>
    // you have the file to process, take care of the line count<BR>
  } else if(file.lastModified() == storedLastModifiedTime) {<BR>
    // reset stored line count<BR>
  }<BR>
}<BR>

这篇关于用Java增量读取日志文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！