Amazon Cloudwatch使用JSON字段记录见解 [英] Amazon Cloudwatch Logs Insights with JSON fields

问题描述

我试图将Logs Insights与其中一个字段中包含JSON的数据一起使用，并解析JSON字段

I am trying to use Logs Insights with data containing JSON in one of the fields, and to parse the JSON fields

当我通过入门代码深入了解数据时，数据如下所示

My data looks like the following when I put it in insights with the starter code

fields @timestamp, @message
| sort @timestamp desc
| limit 25

如何轻松地在嵌套的JSON中提取path变量以对其进行聚合?通过查看一些文档，我认为@message.path可以工作，但事实并非如此.有谁成功解析了Insights中的JSON日志

How can I easily extract the path variable in my nested JSON to perform aggregations on it ? By looking at some documentation, I thought @message.path would work but it does not seem so. Has anyone successfully interpreted JSON logs in Insights

我的数据示例

#
@timestamp
@message
1
2018-12-19 23:42:52.000
I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"user,tags,promotions,company_sector,similar_professionals.tags,similar_professionals.user","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@logStream  i-05d1d61ab853517a0
@message  I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@timestamp  1545262972000
2
2018-12-19 23:42:16.000
I, [2018-12-19T23:42:16.723472 #851] INFO -- : [ea712503-eb86-4a6e-ab38-ddbcd6c2b4d0] {"method":"GET","path":"/api/v1/heartbeats/new","format":"json","controller":"API::V1::Public::HeartbeatsController","action":"new","status":201,"duration":9.97,"view":3.2,"time":"2018-12-19T23:42:16.712+00:00","params":{"format":"json","compress":false},"@timestamp":"2018-12-19T23:42:16.722Z","@version":"1","message":"[201] GET /api/v1/heartbeats/new (API::V1::Public::HeartbeatsController#new)"}

推荐答案

基于 @pyb见解，我能够使用parse @message '"path":"*"' as path从@message中的任何位置提取路径.

Building on @pyb insights, I was able to use parse @message '"path":"*"' as path to extract the path from any place in the @message.

您可以继续通过管道另一个parse @message '"method":"*"' as method来获取方法，而不必担心排序，因为这是在@message

You can go on to get your method by piping another parse @message '"method":"*"' as method without concern for ordering as it is a second global plain text search on @message

如果您的@message是:

I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}

使用:

parse @message '"path":"*"' as path | parse @message '"method":"*"' as method

将在以下字段中显示:path = '/api/v1/professionals/ID'和method = 'GET'

will result in the fields: path = '/api/v1/professionals/ID' and method = 'GET'

请注意，这仍然只是字符串解析，因此，它没有嵌套键的概念，例如params.format不会找到json，但是只要没有其他<format字符串，位于您@message中的任何位置.

Note that this is still simply string parsing and as such, it has no concept of nested keys like params.format would not find json, however using just format would, so long as there wasn't another format string anywhere in your @message.

还请注意，这是针对Insights未在消息中发现您的JSON的情况.我相信这就是@pyb在此答案中所指的情况.使用以下格式也无法找到我的日志

Also note that this is for the case where Insights is not discovering your JSON in the message. I belive this is the case that @pyb was referring to in this answer. My logs aren't being discovered either using the following format

info - Request: {"method":"POST","path":"/auth/login/","body":{"login":{"email":"[e-mail]","password":"********"}},"uuid":"36d76df2-aec4-4549-8b73-f237e8f14e23","ip":"*.*.*.*"}

这篇关于Amazon Cloudwatch使用JSON字段记录见解的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！