更正输入日志的正则表达式 [英] Correct regular expression for the input log
问题描述
输入日志如下所示,其中包含"|"数据精打细算.数据包含id |类型要求|响应
Input log looks like this, which contains data which are "|" sperated. The data contains id | type | request | response
110000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
210000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
340000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
450000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
590000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
所需的输出:
第一个日志:
id- 110000
type-read
request-<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com">
<soapenv:Header/>
<soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>iOBD</serviceRequester>
<lineOfBusiness>mobility</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>911425152231426</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>
response-<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["Micromax"]</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>msisdn</attributeName>
<attributeValue>"Maharashtra"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imei</attributeName>
<attributeValue>"917756870222"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>imsi</attributeName>
<attributeValue>"911425152231426"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_vendor</attributeName>
<attributeValue>"404909092353805"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_type</attributeName>
<attributeValue>"E311"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>device_version</attributeName>
<attributeValue>"1"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>g3</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>mms</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>gprs</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>streaming</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>ota</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>wap</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>os</attributeName>
<attributeValue>"Google"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>version</attributeName>
<attributeValue>"4.4.2"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>camera</attributeName>
<attributeValue>"Yes"</attributeValue>
</mapEntry>
<mapEntry>
<attributeName>apn</attributeName>
<attributeValue>""AIRTELGPRS.COM,AIRTELMMS.COM""</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
第二个日志:
id - 210000
type - read
request -
response -
与"n"号类似.日志
使用的配置文件:
input {
file {
path => "/opt/test5/practice_new/final_xml.dat"
start_position => "beginning"
codec => multiline {
pattern => "^%{NUMBER:method_id}\|%{DATA:method_type}\|<soapenv:Envelope>"
negate => true
what => previous
}
}
}
filter {
grok {
match => [ "message", "(?m)^(?<method_id>\d+)\|(?<method_type>\w+)\|(?<request><soapenv:Envelope>.*?</soapenv:Envelope>)\|(?<response><soapenv:Envelope>.*?</soapenv:Envelope>)" ]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "final"
}
stdout {}
}
我尝试在Grok中使用正则表达式,但是当前表达式不适用于输入日志.
I tried using the regular expression in Grok but the current one is not working for the input logs.
请帮助我提供正则表达式.
please help me with the regular expression.
推荐答案
您当前使用的正则表达式为(?m)^(?<method_id>\d+)\|(?<method_type>\w+)\|(?<request><soapenv:Envelope>.*?</soapenv:Envelope>)\|(?<response><soapenv:Envelope>.*?</soapenv:Envelope>)
,并且如果它们以<soapenv:Envelope>
开头并以<结尾,则只能解析出第三列和第四列. c2>之间有|
.
The regex you currently are using is (?m)^(?<method_id>\d+)\|(?<method_type>\w+)\|(?<request><soapenv:Envelope>.*?</soapenv:Envelope>)\|(?<response><soapenv:Envelope>.*?</soapenv:Envelope>)
, and it can only parse out the 3rd and 4th columns if they start with <soapenv:Envelope>
and end with </soapenv:Envelope>
having |
in between.
似乎您需要一个正则表达式来将第3列标识为除|
以外的任何字符的序列,第4列应将除|
以外的任意数量的字符划分为新字符,直到换行符为止,后跟1个或多个数字,然后按|
.
It seems you need a regex that will identify the 3rd column as a sequence of any chars other than |
and the 4th column should gran any number of chars other than |
up to the newline followed with 1 or more digits and then |
.
使用
(?m)^(?<method_id>\d+)\|(?<method_type>\w+)\|(?<request>[^|]*)\|(?<response>[^|\n]*(?:\n(?!\d+\|)[^|\n]*)*)
请参见 regex演示.
详细信息
-
(?m)
-使.
与换行符匹配的Ruby修饰符 -
^
-一行的开头 -
(?<method_id>\d+)
-组方法":一位或多位数字 -
\|
-管道字符 -
(?<method_type>\w+)
-组"method_type":一个或多个字母,数字或_
-
\|
-管道 -
(?<request>[^|]*)
-组请求":|
以外的任何0+个字符
-
\|
-管道 -
(?<response>[^|\n]*(?:\n(?!\d+\|)[^|\n]*)*)
-组响应":-
[^|\n]*
-除|
和LF(换行符)之外的任何0+个字符 -
(?:\n(?!\d+\|)[^|\n]*)*
-0+次出现:-
\n
-换行符 -
(?!\d+\|)
-后跟1个数字+|
-
[^|\n]*
-除|
和LF(换行符)之外的任何0+个字符
(?m)
- the Ruby modifier that makes.
match line break chars^
- start of a line(?<method_id>\d+)
- Group "method": one or more digits\|
- a pipe char(?<method_type>\w+)
- Group "method_type": one or more letters, digits or_
\|
- a pipe(?<request>[^|]*)
- Group "request": any 0+ chars other than|
\|
- a pipe(?<response>[^|\n]*(?:\n(?!\d+\|)[^|\n]*)*)
- Group "response":[^|\n]*
- any 0+ chars other than|
and LF (newlines)(?:\n(?!\d+\|)[^|\n]*)*
- 0+ occurrences of:\n
- a newline(?!\d+\|)
- not followed with 1+ digits +|
[^|\n]*
- any 0+ chars other than|
and LF (newlines)
这篇关于更正输入日志的正则表达式的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
-
-