Logstash文件输入插件 [英] Logstash file input plugin

问题描述

当前，我正在使用文件输入插件来查看日志存档，但是文件输入插件不是我的正确解决方案，因为文件输入插件固有地希望该文件是事件流，而不是静态文件.现在，这给我造成了很多问题，因为我的日志存档中有100,000多个日志文件，而logstash打开了所有这些永不更改的文件的句柄.

Currently I am using file input plugin to go over my log archive but file input plugin is not the right solution for me because file input plugin inherently expects that file is stream of events and not as a static file. Now, this is causing a great deal of problem for me because my log archive has a 100,000 + log files and I logstash opens a handle on all these files which are never going to change.

我面临以下问题

1)Logstash失败，并出现 SO 中提到的问题
2)由于有许多打开的文件句柄，日志归档存储变得非常缓慢.

1) Logstash fails with problem mentioned in SO
2) With those many open file handles log archival storage is getting very slow.

有人知道让logstash知道静态地对待文件还是一旦处理了文件就不保留文件句柄的方法.

Does anybody know a way to let logstash know that treat files statically or once a file is processed do not keep file handle on it.

在logstash Jira错误中，有人告诉我写一些自己的插件其他建议对我没有多大帮助.

In logstash Jira bug, I was told to write my own plugin with some other suggestions which won't help me much.

推荐答案

Logstash文件输入可以处理静态文件.您需要添加此配置

Logstash file input can process static file. You need to add this configuration

file {
     path => "/your/logs/path"
     start_position => "beginning"
}

添加start_position后，logstash从头开始读取文件.请参阅此处以获取更多信息.请记住，

After adding the start_position, logstash reads the file from the beginning. Please refer here for more information. Remember that

此选项仅修改文件是新文件且之前未曾出现过的首次联系"情况.如果以前已经看过文件，则此选项无效.

this option only modifies "first contact" situations where a file is new and not seen before. If a file has already been seen before, this option has no effect.

否则，您已将sincedb_path设置为/dev/null.

Otherwise you have set your sincedb_path to /dev/null .

对于第一个问题，我在评论中有答复.请尝试添加打开的最大文件数. 对于我的建议，您可以尝试编写一个脚本，将日志文件复制到logstash监视器路径，然后将其不断移出.您必须估计logstash处理日志文件的时间.

For the first question, I have answer in the comment. Please try to add the maximum file opened. For my suggestion, You can try to write a script copy the log file to the logstash monitor path and move it out constantly. You have to estimate the time that logstash process a log file.

这篇关于Logstash文件输入插件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！