在 Java 中执行 PigServer 时出错 [英] Error executing PigServer in Java

问题描述

我正在尝试从我的 Java 机器远程运行 Pig 脚本，为此我编写了以下代码

代码:

import java.io.IOException;导入 java.util.Properties;导入 org.apache.pig.ExecType;导入 org.apache.pig.PigServer;导入 org.apache.pig.backend.executionengine.ExecException;公共类 Javapig{公共静态无效主(字符串 [] args){尝试 {Properties props = new Properties();props.setProperty("fs.default.name", "hdfs://hdfs://192.168.x.xxx:8022");props.setProperty("mapred.job.tracker", "192.168.x.xxx:8021");PigServer pigServer = new PigServer(ExecType.MAPREDUCE, props);运行IdQuery(猪服务器，事实")；}捕获(异常 e){System.out.println(e);}}public static void runIdQuery(PigServer pigServer, String inputFile) 抛出 IOException {pigServer.registerQuery("A = load '" + inputFile + "' using org.apache.hive.hcatalog.pig.HCatLoader();");pigServer.registerQuery("B = FILTER A by category == 'Aller';");pigServer.registerQuery("DUMP B;");System.out.println("完成");}}

但在执行时出现以下错误.

错误

ERROR 4010:在类路径中找不到 hadoop 配置(在类路径中找不到 hadoop-site.xml 和 core-site.xml).

我不知道我做错了什么.

解决方案

好吧，自我描述错误...

<块引用>

在类路径中没有找到 hadoop-site.xml 和 core-site.xml

您的应用程序的类路径中需要这两个文件.

理想情况下，您可以从 $HADOOP_CONF_DIR 文件夹中获取这些文件，然后将它们复制到 Java 的 src/main/resources 中，假设您有一个 Maven 结构>

此外，对于这些文件，您应该为 Hadoop 使用 Configuration 对象

PigServer(ExecType execType, org.apache.hadoop.conf.Configuration conf)

I am trying to run pig scripts remotely from my java machine, for that i have written below code

code:

import java.io.IOException;
import java.util.Properties;
import org.apache.pig.ExecType;
import org.apache.pig.PigServer;
import org.apache.pig.backend.executionengine.ExecException;

public class Javapig{ 
public static void main(String[] args) {
try {
    Properties props = new Properties();
    props.setProperty("fs.default.name", "hdfs://hdfs://192.168.x.xxx:8022");
    props.setProperty("mapred.job.tracker", "192.168.x.xxx:8021");

    PigServer pigServer = new PigServer(ExecType.MAPREDUCE, props);
    runIdQuery(pigServer, "fact");
    }
    catch(Exception e) {
        System.out.println(e);
    }
 }
public static void runIdQuery(PigServer pigServer, String inputFile) throws IOException {
    pigServer.registerQuery("A = load '" + inputFile + "' using org.apache.hive.hcatalog.pig.HCatLoader();");
    pigServer.registerQuery("B = FILTER A by category == 'Aller';");
    pigServer.registerQuery("DUMP B;");
    System.out.println("Done");
 }
}

but while executing i am getting below error.

Error

ERROR 4010: Cannot find hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml was found in the classpath).

I don't know what am i doing wrong.

解决方案

Well, self describing error...

neither hadoop-site.xml nor core-site.xml was found in the classpath

You need both of those files in the classpath of your application.

You ideally would get those from your $HADOOP_CONF_DIR folder, and you would copy them into your Java's src/main/resources, assuming you have a Maven structure

Also, with those files, you should rather use a Configuration object for Hadoop

PigServer(ExecType execType, org.apache.hadoop.conf.Configuration conf)

这篇关于在 Java 中执行 PigServer 时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！