计算事件之间的时间 [英] Calculating time between events
问题描述
filebeat - > logstash - >弹性搜索 - > kibana
因此，我现在有这些事件：
@timestamp messageId事件
2016年5月19日，02：55：29.003 00e02f2f-32d5-9509-870a-f80e54dc8775系统1输入
2016年5月19日，02： 55：29.200 00e02f2f-32d5-9509-870a-f80e54dc8775 system1Exit
2010年5月19日，02：55：29.205 00e02f2f-32d5-9509-870a-f80e54dc8775 system2Enter
2016年5月19日，02：55：29.453 00e02f2f -32d5-9509-870a-f80e54dc8775 system2Exit
我想生成一个报告（理想情况下是堆栈条或列）：
messageId in1：1> 2：in2
00e02f2f- 32d5-9509-870a-f80e54dc8775 197：5：248
最好的方法是什么？ Logstash过滤器？ kibana计算字段？
您可以使用Logstash 聚合过滤器，但是，你会有大幅重新实施 已经过了过滤器已经这样做了，所以这可以是一个耻辱吧？
我们再使用Logstash < a href =https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html =noreferrer> 聚合过滤器和 过滤器过滤器。后者用于测量每个阶段的时间，前者用于将所有时间信息聚合到最后一个事件。
旁注：您可能想重新考虑您的时间戳格式使其更适合于解析。我已经将它们转换为ISO 8601，以便更容易解析，但是可以随意滚动自己的正则表达式。
所以我从以下日志开始： / p>
/-19T02：55：29.003 00e02f2f-32d5-9509-870a-f80e54dc8775 system1Enter
drawings- 19T02：55：29.200 00e02f2f-32d5-9509-870a-f80e54dc8775 system1Exit
籍-19T02：55：29.205 00e02f2f-32d5-9509-870a-f80e54dc8775 system2Enter
IBUS19T02：55： 29.453 00e02f2f-32d5-9509-870a-f80e54dc8775 system2Exit
首先我使用三个已过去过滤器（每个阶段一个 in1 ， 1-> 2 和 in2 ），然后三个聚合过滤器，以收集所有的时间信息。看起来像这样：
filter {
grok {
match => [message，％{TIMESTAMP_ISO8601：timestamp}％{UUID：messageId}％{WORD：event}]
add_tag => [％{event}]
}
date {
match => [timestamp，ISO8601]
}
＃测量系统1的执行时间
已过去{
unique_id_field => messageId
start_tag => system1Enter
end_tag => system1Exit
new_event_on_match => true
add_tag => [in1]
}
＃测量系统2的执行时间
已过去{
unique_id_field => messageId
start_tag => system2Enter
end_tag => system2Exit
new_event_on_match => true
add_tag => [in2]
}
＃测量系统1和系统2之间的时间
已过去{
unique_id_field => messageId
start_tag => system1Exit
end_tag => system2Enter
new_event_on_match => true
add_tag => [1> 2]
}
＃记录system1
的执行时间，如果[tags]中的in1和[tags] {
聚合{
task_id => ％{messageId}
code => map ['report'] = [（event ['elapsed_time'] * 1000）.to_i]
map_action => create
}
}
＃记录system1和system2之间的时间
如果[tags]中的1-&2和[tags] {
aggregate {
task_id => ％{messageId}
code => map ['report']<<（event ['elapsed_time'] * 1000）.to_i
map_action => 更新
}
}
＃记录system2
的执行时间，如果[tags]中的in2和[tags] {
aggregate {
task_id => ％{messageId}
code => map ['report']< {（event ['elapsed_time'] * 1000）.to_i; event ['report'] = map ['report']。join（'：'）
map_action => update
end_of_task => true
}
}
}
在前两个事件你会得到一个这样的新事件，这表明197ms已经花在system1中了：
{
@timestamp=> thai-21T04：20：51.731Z，
tags=> [已过，已经_匹配，in1]，
elapsed_time=> 0.197，
messageId=> 00e02f2f-32d5-9509-870a-f80e54dc8775，
elapsed_timestamp_start=> 2016-05ㄧ-19 00 00 00：：：event event event event event event event event event event event event event event event event event event event event event event event event event event event event event event event event event event event event event你会收到一个这样的事件，它显示了在系统1和系统2之间花费的时间，即5ms：
@timestamp=> thai-21T04：20：51.734Z，
tags=> [已过，已经_匹配，1> 2]，
elapsed_time=> 0.005，
messageId=> 00e02f2f-32d5-9509-870a-f80e54dc8775，
elapsed_timestamp_start=> 嗯〜19：00：55：29.200Z
}
第四个事件之后你会得到一个这样的新事件，它显示了在system2中花费了多少时间，即248ms。该事件还包含报告字段，其中包含消息的所有时间信息
{
@timestamp=> thai-21T04：20：51.736Z，
tags=> [已过，已经_匹配，in2]，
elapsed_time=> 0.248，
messageId=> 00e02f2f-32d5-9509-870a-f80e54dc8775，
elapsed_timestamp_start=> 图19：00：55：29.205Z
report=> 197：5：248
}
I have a message that flows through several systems, each system logs message entry and exit with a timestamp and a uuid messageId. I'm ingesting all logs through:
filebeat --> logstash --> elastic search --> kibana
As a result I now have these events:
@timestamp messageId event
May 19th 2016, 02:55:29.003 00e02f2f-32d5-9509-870a-f80e54dc8775 system1Enter
May 19th 2016, 02:55:29.200 00e02f2f-32d5-9509-870a-f80e54dc8775 system1Exit
May 19th 2016, 02:55:29.205 00e02f2f-32d5-9509-870a-f80e54dc8775 system2Enter
May 19th 2016, 02:55:29.453 00e02f2f-32d5-9509-870a-f80e54dc8775 system2Exit
I would like to produce a report (ideally a stacked bar or column) of time spent in each system:
messageId in1:1->2:in2
00e02f2f-32d5-9509-870a-f80e54dc8775 197:5:248
What is the best way to do this? Logstash filters? kibana calculated fields?
解决方案 You can achieve this with the Logstash aggregate filter only, however, you'd have to substantially re-implement what the elapsed filter already does, so that'd be a shame, right?
Let's then use a mix of the Logstash aggregate filter and the elapsed filter. The latter is used to measure the time of each stage and the former is used to aggregate all the timing information into the last event.
Side note: you might want to rethink your timestamp format to make it something more standard for parsing. I've transformed them to ISO 8601 to make it easier to parse, but feel free to roll your own regex.
So I'm starting from the following logs:
2016-05-19T02:55:29.003 00e02f2f-32d5-9509-870a-f80e54dc8775 system1Enter
2016-05-19T02:55:29.200 00e02f2f-32d5-9509-870a-f80e54dc8775 system1Exit
2016-05-19T02:55:29.205 00e02f2f-32d5-9509-870a-f80e54dc8775 system2Enter
2016-05-19T02:55:29.453 00e02f2f-32d5-9509-870a-f80e54dc8775 system2Exit
First I'm using three elapsed filters (one for each stage in1, 1->2 and in2) and then three aggregate filters in order to gather all the timing information. It looks like this:
filter {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{UUID:messageId} %{WORD:event}"]
add_tag => [ "%{event}" ]
}
date {
match => [ "timestamp", "ISO8601"]
}
# Measures the execution time of system1
elapsed {
unique_id_field => "messageId"
start_tag => "system1Enter"
end_tag => "system1Exit"
new_event_on_match => true
add_tag => ["in1"]
}
# Measures the execution time of system2
elapsed {
unique_id_field => "messageId"
start_tag => "system2Enter"
end_tag => "system2Exit"
new_event_on_match => true
add_tag => ["in2"]
}
# Measures the time between system1 and system2
elapsed {
unique_id_field => "messageId"
start_tag => "system1Exit"
end_tag => "system2Enter"
new_event_on_match => true
add_tag => ["1->2"]
}
# Records the execution time of system1
if "in1" in [tags] and "elapsed" in [tags] {
aggregate {
task_id => "%{messageId}"
code => "map['report'] = [(event['elapsed_time']*1000).to_i]"
map_action => "create"
}
}
# Records the time between system1 and system2
if "1->2" in [tags] and "elapsed" in [tags] {
aggregate {
task_id => "%{messageId}"
code => "map['report'] << (event['elapsed_time']*1000).to_i"
map_action => "update"
}
}
# Records the execution time of system2
if "in2" in [tags] and "elapsed" in [tags] {
aggregate {
task_id => "%{messageId}"
code => "map['report'] << (event['elapsed_time']*1000).to_i; event['report'] = map['report'].join(':')"
map_action => "update"
end_of_task => true
}
}
}
After the first two events, you'll get a new event like this, which shows that 197ms have been spent in system1:
{
"@timestamp" => "2016-05-21T04:20:51.731Z",
"tags" => [ "elapsed", "elapsed_match", "in1" ],
"elapsed_time" => 0.197,
"messageId" => "00e02f2f-32d5-9509-870a-f80e54dc8775",
"elapsed_timestamp_start" => "2016-05-19T00:55:29.003Z"
}
After the third event, you'll get an event like this, which shows how much time is spent between system1 and system2, i.e. 5ms:
{
"@timestamp" => "2016-05-21T04:20:51.734Z",
"tags" => [ "elapsed", "elapsed_match", "1->2" ],
"elapsed_time" => 0.005,
"messageId" => "00e02f2f-32d5-9509-870a-f80e54dc8775",
"elapsed_timestamp_start" => "2016-05-19T00:55:29.200Z"
}
After the fourth event, you'll get a new event like this one, which shows how much time was spent in system2, i.e. 248ms. That event also contains a report field with all the timing information of the message
{
"@timestamp" => "2016-05-21T04:20:51.736Z",
"tags" => [ "elapsed", "elapsed_match", "in2" ],
"elapsed_time" => 0.248,
"messageId" => "00e02f2f-32d5-9509-870a-f80e54dc8775",
"elapsed_timestamp_start" => "2016-05-19T00:55:29.205Z"
"report" => "197:5:248"
}
这篇关于计算事件之间的时间的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
