PIG中的Elephant-Bird UDF中的JSON数组字段处理 [英] JSON Array field handling in Elephant-Bird UDF in PIG

问题描述

有关PIG中JSON处理的快速问题.

A quick question on the JSON handling in PIG.

我尝试了一些名为Elephant-Bird的JsonLoader来加载和处理JSON数据，如下所示:

I tried some JsonLoader called Elephant-Bird to load and handle JSON data like the followings:

{
   "SV":1,
   "AD":[
      {
         "ID":"46931606",
         "C1":"46",
         "C2":"469",
         "ST":"46931",
         "PO":1
      },
      {
         "ID":"46721489",
         "C1":"46",
         "C2":"467",
         "ST":"46721",
         "PO":5
      }
   ]
}

该加载器对于简单字段运行良好，但不适用于任何数组字段.我不知道如何使用此UDF或以任何其他方式访问数组(上面的"AD"字段中)的元素?请指教.

The loader works well for simple fields but it doesn't work well for any array field. I don't know how I can access elements in the array ("AD" field above) with this UDF or in any other way? Please advise.

推荐答案

您应使用-nestedLoad参数，如下所示:

You should use -nestedLoad param like this:

a = load 'input' using com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS (json:map[]).

然后使用以下代码:

b = FOREACH a GENERATE (json#'AD') as AD:bag{t:Tuple(m:map[])};

然后，您的json数组将成为bag数据类型.您可以将其展平以获取元组.

Then your json array become a bag datatype. You can flatten it to get tuple.

c = FOREACH b GENERATE FLATTEN(AD);
d = FOREACH c GENERATE AD::m#ID AS ID, AD::m#C1 AS C1, AD::m#C2 AS C2, AD::m#ST AS ST, AD::m#PO AS PO

这时，您将获得模式为(ID:bytearray，C)的元组数据类型

At this time, you will get the tuple data type which the schema is (ID:bytearray, C)

这篇关于PIG中的Elephant-Bird UDF中的JSON数组字段处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！