如何使用Pig从列中解析JSON字符串 [英] How to parse a JSON string from a column with Pig

问题描述

我有tsv日志文件，其中一列由json字符串填充。

I have tsv log files where a column is populated by a json string.

我想在 Pig 脚本中使用 JsonLoader 解析该列。
我看到很多例子，其中 JsonLoader 用于每行只有一个json字符串的情况。

I want to parse that column with JsonLoader in a Pig script. I saw many examples where JsonLoader is used in cases where each row is only a json string. I have other columns I want to skip and I don't know how to do that.

该文件如下所示：

foo    bar    {"version":1; "type":"an event"; "count": 1}
foo    bar    {"version":1; "type":"another event"; "count": 1}

我该怎么做？

推荐答案

您正在寻找Elephant Bird提供的JsonStringToMap UDF： https://github.com/kevinweil/elephant-bird/search?q=JsonStringToMap&ref=cmdform

You are looking for the JsonStringToMap UDF provided in Elephant Bird: https://github.com/kevinweil/elephant-bird/search?q=JsonStringToMap&ref=cmdform

示例文件：

foo     bar     {"version":1, "type":"an event", "count": 1}
foo     bar     {"version":1, "type":"another event", "count": 1}

猪脚本：

REGISTER /path/to/elephant-bird.jar;
DEFINE JsonStringToMap com.twitter.elephantbird.pig.piggybank.JsonStringToMap();
raw = LOAD '/tmp/file.tsv' USING PigStorage('\t') AS (col1:chararray,col2:chararray,json_string:chararray);
parsed = FOREACH raw GENERATE col1,col2,JsonStringToMap(json_string);
ILLUSTRATE parsed; -- Just to show the output

预处理（JSON as chararray / string）：

Pre-processing (JSON as chararray/string):

-------------------------------------------------------------------------------------------------------
| raw     | col1:chararray    | col2:chararray    | json_string:chararray                             | 
-------------------------------------------------------------------------------------------------------
|         | foo               | bar               | {"version":1, "type":"another event", "count": 1} | 

后处理（JSON as map）：

---

Post-processing (JSON as map):

-------------------------------------------------------------------------------------------------
| parsed     | col1:chararray    | col2:chararray    | json:map(:chararray)                     | 
-------------------------------------------------------------------------------------------------
|            | foo               | bar               | {count=1, type=another event, version=1} | 
-------------------------------------------------------------------------------------------------

以下是两天前提出的问题：

Here's the same question asked just two days ago: How do you decode JSON in Pig that comes from a column?

这篇关于如何使用Pig从列中解析JSON字符串的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！