如何使用 Pig 从列中解析 JSON 字符串 [英] How to parse a JSON string from a column with Pig
问题描述
我有 tsv 日志文件,其中一列由 json 字符串填充.
I have tsv log files where a column is populated by a json string.
我想在 Pig 脚本中使用 JsonLoader 解析该列.我看到了很多例子,其中 JsonLoader 用于每行只是一个 json 字符串的情况.我还有其他专栏想跳过,但不知道该怎么做.
I want to parse that column with JsonLoader in a Pig script. I saw many examples where JsonLoader is used in cases where each row is only a json string. I have other columns I want to skip and I don't know how to do that.
文件如下所示:
foo bar {"version":1; "type":"an event"; "count": 1}
foo bar {"version":1; "type":"another event"; "count": 1}
我该怎么做?
推荐答案
您正在寻找 Elephant Bird 中提供的 JsonStringToMap UDF:https://github.com/kevinweil/elephant-bird/search?q=JsonStringToMap&ref=cmdform
You are looking for the JsonStringToMap UDF provided in Elephant Bird: https://github.com/kevinweil/elephant-bird/search?q=JsonStringToMap&ref=cmdform
示例文件:
foo bar {"version":1, "type":"an event", "count": 1}
foo bar {"version":1, "type":"another event", "count": 1}
猪脚本:
REGISTER /path/to/elephant-bird.jar;
DEFINE JsonStringToMap com.twitter.elephantbird.pig.piggybank.JsonStringToMap();
raw = LOAD '/tmp/file.tsv' USING PigStorage('\t') AS (col1:chararray,col2:chararray,json_string:chararray);
parsed = FOREACH raw GENERATE col1,col2,JsonStringToMap(json_string);
ILLUSTRATE parsed; -- Just to show the output
预处理(JSON 作为字符数组/字符串):
Pre-processing (JSON as chararray/string):
-------------------------------------------------------------------------------------------------------
| raw | col1:chararray | col2:chararray | json_string:chararray |
-------------------------------------------------------------------------------------------------------
| | foo | bar | {"version":1, "type":"another event", "count": 1} |
后处理(JSON 作为地图):
Post-processing (JSON as map):
-------------------------------------------------------------------------------------------------
| parsed | col1:chararray | col2:chararray | json:map(:chararray) |
-------------------------------------------------------------------------------------------------
| | foo | bar | {count=1, type=another event, version=1} |
-------------------------------------------------------------------------------------------------
这是两天前提出的相同问题:如何在来自列的 Pig 中解码 JSON?
Here's the same question asked just two days ago: How do you decode JSON in Pig that comes from a column?
这篇关于如何使用 Pig 从列中解析 JSON 字符串的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!