在日志中快速搜索 [英] Fast Search in Logs

问题描述

我遇到的问题与另一篇文章中所述的问题相同。我的应用程序的日志文件是巨大的（〜1GB），grep是繁琐的用来关联日志文件的信息。现在我使用''较少''工具，但它也比我想要的慢。

I have the same problem as somebody described in another post. My application's log files are huge (~1GB), and grep is tedious to use to correlate information from the log files. Right now I use the ''less'' tool, but it is also slower than I would like.

我想加快搜索速度。有以下方法来做到这一点：首先，生成XML的日志，并使用一些XML搜索工具。我不知道使用XML搜索将获得多少加速（不太多，我猜，因为非索引文件搜索仍然需要时间）。

I am thinking of speeding up the search. There are the following ways to do this: first, generate logs in XML and use some XML search tool. I am not sure how much speedup will be obtained using XML search (not much I guess, since non-indexed file search will still take ages).

其次，使用 XML数据库。这将是更好的，但我没有太多的背景。

Second, use an XML database. This would be better, but I don't have much background here.

第三，使用（非XML）数据库。这将是有点冗长乏味，因为表模式必须写入（上面的第二个选项是否已经做了？）。我也预见到架构在开始改变很多，以包括常见的用例。理想情况下，我想要比用于存储日志的完整数据库更轻的东西。

Third, use a (non-XML) database. This would be somewhat tedious since the table schema has to be written (has it to be done for second option above too?). I also foresee the schema to change a lot at the start to include common use cases. Ideally, I would like something lighter than a full-fledged database for storing the logs.

第四，使用lucene。它似乎适合的目的，但有一个简单的方法来指定当前用例的索引？例如，每当你看到迭代一词，我想说index。

Fourth, use lucene. It seems to fit the purpose, but is there a simple way to specify the indexes for the current use case? For example, I want to say "index whenever you see the word 'iteration'".

你的意见是什么？

推荐答案

问题是使用XML会使你的日志文件更大
我建议拆分日志文件按日期或行
否则使用基于文件的数据库引擎，如 sqlite

The problem is using XML will make your log file even bigger I would suggest either splitting up your log files by date or lines otherwise use file based database engines such as sqlite

这篇关于在日志中快速搜索的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！