如何根据日期提交过滤logstash的输入数据? [英] How to filter input data of logstash based on date filed?
问题描述
这里是我的微博输入tweets
here is my twitter input tweets
"_source": {
"created_at": "Wed Aug 10 06:42:48 +0000 2016",
"id": 763264318242783200,
"timestamp_ms": "1470811368891",
"@version": "1",
"@timestamp": "2016-08-10T06:42:48.000Z"
}
和我的logstash配置文件,其中包括twitter输入插件
过滤器和输出
and my logstash config file which include twitter input plugin
filter and output
input {
twitter {
consumer_key => "lvvoeonCRBOHsLAoTPbion9sK"
consumer_secret => "GNHOFzErJhuo0bNq38JUs7xea2BOktMiLa7tunoGwP0oFKCHrY"
oauth_token => "704578110616936448-gfeSklNrITu7fHIZgjw3nwoZ1S0l0Jl"
oauth_token_secret => "IHiyRJRN09jjdUTGrnesALw4DRle35WyX7pdnI3CtEnJ5"
keywords => [ "afghanistan", "TOLOnews", "kabul", "police"]
full_tweet => true
}
}
filter {
date {
match => ["timestamp" , "MMM d YYY HH:mm:ss", "ISO8601"]
}
}
output {
stdout { codec => dots }
elasticsearch {
hosts => "10.20.1.123"
index => "twitter_news"
document_type => "tweets"
}
}
我想要得到新的推文例如今天的日期是2016-11-16,那么我只想得到有 @ timestamp = 2016-11-16
而不是 @ timestamp = 2016-11-15
或过去几天的tweets,但是通过这个配置,我也得到了过去的推文,任何一个帮助我怎么做?
I want to just get new tweets for example today date is 2016-11-16, then I just want to get tweets that have @timestamp= 2016-11-16
not @timestamp= 2016-11-15
or past days tweets, but with this configuration i get past tweets as well, any one help me to how do this ?
推荐答案
这里的想法是在logstash config中使用ruby代码。
我建议使用 timestamp_ms
来比较日期。
the idea here is to use ruby code in logstash config.
I propose to use timestamp_ms
for comparing date.
- 首先需要将
timestamp_ms
转换为整数 -
比较时间戳
这是一个例子:
- First need to convert
timestamp_ms
to integer - Add today timestamp in ms with ruby
Compare timestamps Here is an example:
mutate {
convert => {
"timestamp_ms" => "integer"
}
}
ruby {
code => "
t = Time.now
today_ymd = t.strftime('%Y%m%d')
today_timestamp_ms = DateTime.parse(today_ymd).to_time.to_i*1000
event['@metadata']['today_timestamp_ms'] = today_timestamp_ms
"
}
if [timestamp_ms] < [@metadata][today_timestamp_ms] {
## past days events
mutate {
add_field => { "test" => "past days events" }
}
} else {
# today events
mutate {
add_field => { "test" => "today events" }
}
}
这篇关于如何根据日期提交过滤logstash的输入数据?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!