如何将数据从应用程序日志文件提取到csv文件 [英] How to extract data from application log file to csv file

问题描述

我在下面的部分中有一个application.log文件

I have a application.log file in that from below section

2020-04-14 01:04:06 INFO  app-proxy:53 - Params : {"file_list":["USER_INFO_1234567.tar.md5"],"conn_id":"6adfda","Token":"vishal.kg","proxy_name":"India-Noida","hash":"sdadjsasdkj"}
2020-04-14 01:24:44 INFO  app-proxy:53 - Params : {"file_list":["ATT_SAMPLE_TERT.tar.md5"],"conn_id":"adfdsfed","Token":"venkat.raj","proxy_name":"India-Noida","hash":"qieuadsjkasdjk"}
2020-04-14 03:16:06 INFO  app-proxy:53 - Params : {"file_list":["KER_SAMPLE_TERT.tar.md5"],"conn_id":"kajdfldk","Token":"ankit.ys","proxy_name":"India-Noida","hash":"asdkfjds"}
2020-04-14 03:18:15 INFO  app-proxy:53 - Params : {"file_list":["DU_SAMPLE_TERT.tgz"],"conn_id":"u9sdf7ds9","Token":"shitha.a","proxy_name":"India-Noida","hash":"wqeirdasjk"}
2020-04-14 04:30:02 INFO  app-proxy:53 - Params : {"file_list":["MBU_SAMAPLE.tar.md5"],"conn_id":"a8df7dsd","Token":"karthi.v","proxy_name":"India-Noida","hash":"odisfdjda"}
2020-04-14 05:22:06 INFO  app-proxy:53 - Params : {"file_list":["PCL_SAMPLE-15637481.tar.md5"],"conn_id":"8adf8das","Token":"b.venkat","proxy_name":"India-Noida","hash":"adfjkds"}

我正在尝试找到Params :部分，因此我需要将Date, Time, file_list, Token, proxy_name提取到csv文件中.

i am trying to find Params : section and from that i need to extract Date, Time, file_list, Token, proxy_name into csv file.

然后我还有另一个user-session.log文件，它包含在下面.

Then i have another user-session.log file and it contain below.

2020-04-14 01:04:07 REMOTE_START null null USER_INFO_1234567.tar.md5 0 0/0
2020-04-14 01:15:18 REMOTE_END null null USER_INFO_1234567.tar.md5 672 7219067/7209967

2020-04-14 01:24:45 REMOTE_START null null ATT_SAMPLE_TERT.tar.md5 0 0/0
2020-04-14 01:34:12 REMOTE_END null null ATT_SAMPLE_TERT.tar.md5 568 52401769/50176769

2020-04-14 03:16:08 REMOTE_START null null KER_SAMPLE_TERT.tar.md5 0 0/0
2020-04-14 03:16:22 REMOTE_END null null KER_SAMPLE_TERT.tar.md5 16 1059346/70840514

2020-04-14 03:18:17 REMOTE_START null null DU_SAMPLE_TERT.tgz 0 0/0
2020-04-14 03:18:18 REMOTE_END null null DU_SAMPLE_TERT.tgz 2 949685/949685

2020-04-14 04:30:04 REMOTE_START null null MBU_SAMAPLE.tar.md55 0 0/0
2020-04-14 04:30:05 REMOTE_END null null MBU_SAMAPLE.tar.md5 2 2857069/2857069

2020-04-14 05:22:12 REMOTE_START null  PCL_SAMPLE-15637481.tar.md5 0 0/0
2020-04-14 05:22:15 REMOTE_END null null PCL_SAMPLE-15637481.tar.md5 9 93829204/93829204

从此REMOTE_START& REMOTE_END数据包含每个用户.我需要将相同的内容与上述csv文件一起提取.根据下面的屏幕截图要求的输出.

From this REMOTE_START & REMOTE_END data contain for each user. I need to extract the same along with above csv file. desired output required as per below screenshot.

任何方向都会有很大帮助.

Any direction would be great helpful.

推荐答案

由于OP更改了Input_file的样本数据，因此现在按其添加解决方案.

Since OP changed Input_file(s) sample data so adding solution as per it now.

awk '
FNR==NR{
  match($0,/{"file_list":\["[^"]*/)
  userData=substr($0,RSTART,RLENGTH)
  sub(/{"file_list":\["/,"",userData)
  match($0,/"Token":"[^"]*/)
  tokenVal=substr($0,RSTART,RLENGTH)
  sub(/.*"/,"",tokenVal)
  match($0,/"proxy_name":"[^"]*/)
  proxyName=substr($0,RSTART,RLENGTH)
  sub(/.*"/,"",proxyName)
  arrayId[userData]=$1 OFS $2 OFS userData OFS tokenVal OFS proxyName
  next
}
!firstOccur[$6]++{
  firstOccurVal[$6]=$2
}
($6 in arrayId){
  Output[$6]=arrayId[$6] OFS firstOccurVal[$6] OFS $2
}
END{
  for(key in Output){
    print Output[key]
  }
}' application.log user-session.log

如果要以美观的格式输出，还可以在上述代码后附加| column -t.

Also append | column -t to above code in case you want output in a nice looking format.

请您试一下.

awk '
match($0,/USERDATA_TMB_[^"]*/){
  userd=substr($0,RSTART,RLENGTH)
  split(userd,array,"_")
  val=array[3]
}
FNR==NR{
  match($0,/"Token":"[^"]*/)
  tVal=substr($0,RSTART,RLENGTH)
  sub(/.*"/,"",tVal)
  match($0,/"proxy_name":"[^"]*/)
  pname=substr($0,RSTART,RLENGTH)
  sub(/.*"/,"",pname)
  arrayId[val]=$1 OFS $2 OFS userd OFS tVal OFS pname
  next
}
!firstOccur[val]++{
  firstOccurVal[val]=$2
}
(val in arrayId){
  Output[val]=arrayId[val] OFS firstOccurVal[val] OFS $2
}
END{
  for(key in Output){
    print Output[key]
  }
}
' application.log user-session.log

这篇关于如何将数据从应用程序日志文件提取到csv文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！