Logstash是否可能将相同内容从日志文件推送到ElasticSearch [英] Is it possible Logstash push same content from log file to ElasticSearch

问题描述

logstash 配置将日志文件设置为输入源，然后将内容发送到 ElasticSearch .

The logstash config sets log files as input source and then sends the content to ElasticSearch.

input 部分如下所示

input{
    file{
        path => "/data/logs/backend.log*"
        start_position => "beginning"
    }
}

然后，日志文件将按大小滚动，这意味着日志文件名首先为 backend.log ，当文件大小达到10M时，将其重命名为 backend.log.1 ，并创建一个新的空backend.log来记录内容.

Then the log file will be rolling by size, which means at first the log file name is backend.log, when the file reaches size 10M, then it is renamed to backend.log.1, and a new empty backend.log is created to log content.

所以问题是 logstash 是否会将内容从 backend.log.1 发送到es服务器?还是 ElasticSearch 能够区分已经收到 backend.log.1 中的内容，尽管这似乎效率不高.

So the question is whether logstash will send the content from backend.log.1 to es server? Or is ElasticSearch able to distinguish that the content from backend.log.1 already received, although this seems to be not efficient.

推荐答案

file 输入文档包含有关如何处理

The file input documentation contains a whole paragraph about how well it handles rotation

此输入检测并处理文件旋转，而不管文件是通过重命名还是通过复制操作旋转.为了支持在旋转发生后一段时间内写入旋转文件的程序，请在文件名模式中同时包含原始文件名和旋转文件名(例如/var/log/syslog和/var/log/syslog.1).观看(路径选项).

File rotation is detected and handled by this input, regardless of whether the file is rotated via a rename or a copy operation. To support programs that write to the rotated file for some time after the rotation has taken place, include both the original filename and the rotated filename (e.g. /var/log/syslog and /var/log/syslog.1) in the filename patterns to watch (the path option).

由于默认为 tail 模式，因此 path 参数应确保使用glob模式来捕获所有文件，就像您所做的一样.所以，您都准备好了.拖尾很开心！

Since the tail mode is the default, your path parameter should make sure to use a glob pattern to catch all files, exactly as you did. So you're all set. Happy tailing!

这篇关于Logstash是否可能将相同内容从日志文件推送到ElasticSearch的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！