如何在猪文件中使用 .jar [英] How to use .jar in a pig file

问题描述

我有两个输入文件 smt.txt 和 smo.txt.jar 文件读取文本文件并根据 java 文件中描述的某些规则拆分数据.而pig文件通过做mapreduce将这些数据放入输出文件中.

I have two input files smt.txt and smo.txt. The jar file reads the text files and split the data according to some rule which is described in java file. And the pig file takes these data put into output files with doing mapreduce.

register 'maprfs:///user/username/fl.jar';
DEFINE FixedLoader fl();

mt = load 'maprfs:///user/username/smt.txt' using FixedLoader('-30','30-33',...........) AS (.........);

mo = load 'maprfs:///user/username/smo.txt*' using FixedLoader('-30','30-33',.....) AS (......);

store mt into 'maprfs:///user/username/mt_out' using JsonStorage();
store mo into 'maprfs:///user/username/mo_out' using JsonStorage();

以及如下所示的部分 java 代码.(我相信方法的内容不是必需的):

and a part of java code like in the following. (The content of methods are not neccessary I believe):

package com.mapr.util;

import org.apache.hadoop.mapreduce.lib.input.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.io.*;
import org.apache.pig.*;
import org.apache.pig.data.*;
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.*;
import java.util.*;
import java.io.*;

public class FixedLoader extends LoadFunc
{

............

}

当我在终端中使用命令pig -x mapreduce sample.pig"运行这个 pig 程序时，我给出了一条错误消息:

When I run this pig program in a teminal with the command "pig -x mapreduce sample.pig", I gave an Error message:

错误 org.apache.pig.tools.grunt.Grunt - 错误 1070:无法使用导入解析 com.mapr.util.FixedLoader:[, org.apache.pig.builtin., org.apache.pig.impl.内置.]

ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve com.mapr.util.FixedLoader using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.]

如何将这些导入到我的项目中，或者是否有任何建议/解决方案来运行此程序?

How can I import these into my project or are there any suggestions/solutions to run this program?

推荐答案

你需要定义FixedLoader及其完整的包名:

You need to define FixedLoader with its full package name:

register 'maprfs:///user/username/fl.jar';
DEFINE FixedLoader com.mapr.util.FixedLoader();
...

还要注册在您的自定义 UDF 中使用的所有 3rd 方依赖项 jar.

Also register all of the 3rd party dependency jars that are used in your custom UDF.

这篇关于如何在猪文件中使用 .jar的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！