FileBeat收集问题 [英] FileBeat harvesting issues

问题描述

我们正在使用ELK来控制程序日志。
在我们的FileBeat配置中，我们从 30个不同的路径中进行收获，该路径包含每秒更新的文件（它仅在prod的机器中每秒更新一次-在其他Dev机器中，我们的日志大大减少了）。
我们的日志文件变旧后才会被删除，我们将停止使用它们（也不会在此修改名称）。
最近，我们发现配置文件（.yml）中来自生产机器的文件中最后路径的日志从未出现在Kibana中。

We are using ELK for controlling our program logs. In our FileBeat config we are harvesting from 30 different paths which contains files that updates every second (it updates every second only in the prod's machines - in the other Dev machines we have significantly less logs). Our log files not get deleted until they getting old and we stop using them (also we don't modify there names) . Lately we found out that the logs from last paths in the configuration file (.yml) from the prod machines is never appearing in the Kibana.

经过调查，我们意识到卡在文件上的FileBeat是第一个路径，似乎从未到达最后一个路径。当我替换开始位置的最后两个路径的位置时，FileBeat开始在此处注册所有日志，并在以后收集它们。

After investigation, we realized that FileBeat stuck on the files is the first's path's and never seem to reach the last one's. When I replace the location of the last two paths to the beginning, FileBeat started to register all the logs there and later on harvest them.

我在文档中查找了FileBeat配置，我看到了close *选项 close_option_config ，似乎是个好主意。但是我还没有做好，我不确定scan_frequency选项的建议时间是多少（目前默认为10s），什么会以最佳方式为我服务。

I looked up in the documentation on the FileBeat configuration and I saw the close* options close_option_config which seem like a good idea. But I didn't managed to get it right yet and I don't sure what is the recommended time for the scan_frequency option (that for now is default of 10s) and what would serve me in the best way.

我尝试将 close_timeout 更改为15s，将 scan_frequency 更改为2m

I tried to change the close_timeout to 15s and the scan_frequency to 2m

      close_timeout: 15s
      scan_frequency: 2m

我想在这里发表一些意见，我应该怎么做才能解决这个问题？我将配置放在这里以供参考，以查看是否遗漏了其他内容。

I would like to here some opinion what can I do to get solve this problem? I put the config here to have some reference and to see if I missed something else.

我的filebeat.yml ：（更改前）

my filebeat.yml: (before changes)

      filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      paths:
        - D:\logs\*\path1\a_*_Pri_app.log.txt
      input_type: log
      document_type: type1
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
    -
      paths:
        - D:\logs\*\path2\b_*_Paths_app.log.txt
      input_type: log
      document_type: type2
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
    -
      paths:
        - D:\logs\*\path3\c_*_R_app.log.txt
      input_type: log
      document_type: path3
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
    -
      paths:
        - D:\logs\*\path4\d_*_d_app.log.txt
        - C:\logs\*\path4\d_*_d_app.log.txt
      input_type: log
      document_type: path4
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after

之后。与上述相同

 paths:
        - D:\logs\*\path27\S.Coordinator_Z.*.log*
        - C:\logs\*\path27\S.Coordinator_Z*.log*
      input_type: log
      document_type: path27
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
    -
      paths:
        - D:\logs\*\path28\d_*_Tr_app.log.txt
        - C:\logs\*\path28\d_*_Tr_app.log.txt
      input_type: log
      document_type: path28
      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after
    -
      paths:
        - D:\logs\*\R1_Output\R*\pid_*_rr_*
      input_type: log
      document_type: path29
      multiline.pattern: '<\?xml version="1\.0" encoding="UTF-8"\?>'
      multiline.negate: true
      multiline.match: after  
    -
      paths:
        - D:\logs\*\R2_Output\R*\pid_*_rr_*
      input_type: log
      document_type: path30
      multiline.pattern: '<\?xml version="1\.0" encoding="UTF-8"\?>'
      multiline.negate: true
      multiline.match: after

      registry_file: "C:/ProgramData/filebeat/registry"

推荐答案

经过长时间的调查，我试图找到与相似的问题>解决方案，然后在dicuss弹性论坛中尝试了我的运气。
我设法解决了这个问题。

After a long investigation when i tried to find a similar problem to what i had with a solution, and after trying my luck in the dicuss elastic forum. I managed to solve this issue.

由于我没有在网络上看到此选项，因此将其放在此处。

Since I didn't see this option in the web i am putting it here.

当同时处理大量打开的文件时，Filebeat收集系统显然具有限制。 （一个已知的问题和弹性团队还提供了许多配置选项，以帮助解决此问题并根据您的需要打扮ELK，例如 config_options ）。
我设法通过打开另外2个Filebeat服务来解决我的问题，我通过以下方式配置了它们的探矿者（A的一个相同示例转到B）：

Filebeat harvesting system apparently has it limit when it comes with dealing with a big scale number of open files in the same time. (a known problem and elastic team also provides bunch of config options to help dealing this issue and costume ELK to your need, e.g config_options). I managed to solve my problem with opening 2 more Filebeat services which i configures their prospectors in the following way(an example of A same goes to B):

paths:
    - D:\logs\*\pid_*_rr_*
  input_type: log
  document_type: A 
  multiline.pattern: '<\?xml version="1\.0" encoding="UTF-8"\?>'
  multiline.negate: true
  multiline.match: after
  close_eof: true

这样，因为相互依赖工作的服务Filebeat一直试图运行

In this way, because the services working interdependently Filebeat keep trying to operate them (and not "stuck" on the first prospectors).

我以这种方式设法使我的收获能力翻了一番。

I manage in this way to double my harvesting abilities.

在Elastic网站上提出讨论：
讨论

posing up the discussion in Elastic website: the discussion

这篇关于FileBeat收集问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！