存储在hdfs中的纱线解析作业日志 [英] Yarn parsing job logs stored in hdfs

问题描述

是否有任何解析器，可用于解析存在于hdfs中的纱线作业日志(jhist文件)中的json，以从中提取信息.

Is there any parser, which I can use to parse the json present in yarn job logs(jhist files) which gets stored in hdfs to extract information from it.

推荐答案

.jhist文件中的第二行是文件中其他json的avro模式.这意味着您可以从jhist文件中创建avro数据. 为此，您可以使用 avro-tools-1.7.7.jar

The second line in the .jhist file is the avro schema for the other jsons in the file. Meaning that you can create avro data out of the jhist file. For this you could use avro-tools-1.7.7.jar

# schema is the second line
sed -n '2p;3q' file.jhist > schema.avsc

# removing the first two lines
sed '1,2d' file.jhist > pfile.jhist

# finally converting to avro data
java -jar avro-tools-1.7.7.jar fromjson pfile.jhist --schema-file schema.avsc > file.avro

您有一个Avro数据，例如，您可以将其导入到Hive表中并对其进行查询.

You've got an avro data, which you can for example import to a Hive table, and make queries on it.

这篇关于存储在hdfs中的纱线解析作业日志的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！