启动Sqoop2服务器时无法找到hadoop配置类（1.99.7） [英] Unable to find hadoop Configuration classes when starting Sqoop2 server (1.99.7)

问题描述

我正在使用带有最新版本的Sqoop2（1.99.7）的Hadoop进行测试，并且在运行sqoop2-server时，出现以下错误：

 线程main中的异常java.lang.NoClassDefFoundError：org / apache / hadoop / conf / Configuration 
 at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler。安全登录（SimpleAuthenticationHandler.java:36）
在org.apache.sqoop.security.AuthenticationManager.initialize（AuthenticationManager.java:98）
在org.apache.sqoop.core.SqoopServer.initialize（SqoopServer。 Java：57）
 at org.apache.sqoop.server.SqoopJettyServer。< init>（SqoopJettyServer.java:67）
 at org.apache.sqoop.server.SqoopJettyServer.main（SqoopJettyServer.java ：177）
 
导致：java.lang.ClassNotFoundException：org.apache.hadoop.conf.Configuration $ b $ java.net.URLClassLoader $ 1.run（URLClassLoader.java:366）
在java.net.URLClassLoader $ 1.run（URLClassLoader.java:355）
在java.se curity.AccessController.doPrivileged（本地方法）$ b $在java.net.URLClassLoader.findClass（URLClassLoader.java:354）
在java.lang.ClassLoader.loadClass（ClassLoader.java:425）
在sun.misc.Launcher $ AppClassLoader.loadClass（Launcher.java:308）
在java.lang.ClassLoader.loadClass（ClassLoader.java:358）
 ... 5 more 
  

我目前的设定：

• 使用最新的 https://hub.docker.com/r/sequenceiq/hadoop Hadoop的-docker / 图片


• 下载最新的Sqoop2二进制文件（ http://sqoop.apache.org/ ）并将它们安装到/ usr / lib / sqoop /
运行的容器中。

• 运行/ usr / lib / sqoop / bin / sqoop2-server start

最好我能想到的是Hadoop类路径并未被Sqoop加载，因为所需的JAR似乎位于/ usr / local / hadoop / shared /*.

我能在网上找到的大多数文档都是在1.99之前.7，但这个版本的一个主要变化是Sqoop服务器从Tomcat移动到Jetty，所以所有的catalina配置选项都是没有意义的。

有人可以帮我吗弄清楚如何让Sqoop服务器运行？

解决方案

嗯，大部分都是这样。

b
$ b

看起来像sqoop.sh脚本根据环境变量设置从类路径加载所有jar。 Docker容器将所有正在查找的环境变量设置为Hadoop安装的根路径，而脚本期望它们指向包含JAR文件的目录。

  HADOOP_PREFIX = / usr / local / hadoop 
 HADOOP_HDFS_HOME = / usr / local / hadoop 
 HADOOP_COMMON_HOME = / usr / local / hadoop 
 HADOOP_YARN_HOME = / usr / local / hadoop 
 HADOOP_CONF_DIR = / usr / local / hadoop / etc / hadoop 
 HADOOP_MAPRED_HOME = / usr / local / hadoop 
  

因此，脚本通过$ HADOOP_HOME目录和随后在脚本中调用的子目录选择此目录。

最后一步是编辑sqoop.properties文件并确保mapreduce配置设置为正确的目录：

  org.apache.sqoop.submission.engine.mapreduce.configuration.directory = / usr / local / hadoop / etc / hadoop 
  
 
 
 
然后服务器启动！
 
 
 
我会在这里留下以防其他人碰到这... ... 
I'm working on testing out using Hadoop with the latest version of Sqoop2 (1.99.7), and when running the sqoop2-server, I get the following error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:36)
at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:98)
at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
at org.apache.sqoop.server.SqoopJettyServer.<init>(SqoopJettyServer.java:67)
at org.apache.sqoop.server.SqoopJettyServer.main(SqoopJettyServer.java:177)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 5 more
My current setup:


Using the latest https://hub.docker.com/r/sequenceiq/hadoop-docker/ image for Hadoop
Downloaded the latest Sqoop2 binaries (http://sqoop.apache.org/) and installed them into a running container at /usr/lib/sqoop/
Run /usr/lib/sqoop/bin/sqoop2-server start


Best I can figure is the Hadoop classpath isn't being loaded by Sqoop, as the required JAR's appear to be located at /usr/local/hadoop/shared/*.

Most of the documentation I've been able to find online is for pre-1.99.7, but one major change in this version is that the Sqoop server moved from Tomcat to Jetty, so all of the catalina configuration options are moot.

Can someone help me figure out how to get Sqoop server to run?
解决方案
Ah, figured it out for the most part.

Looks like the sqoop.sh script loads all of the jars from the classpath based on the environment variables set.  The docker container has all of the environment variables that it's looking for set to the root path of the Hadoop installation, while the script is expecting them to be pointing to the directories containing the JAR files.
HADOOP_PREFIX=/usr/local/hadoop
HADOOP_HDFS_HOME=/usr/local/hadoop
HADOOP_COMMON_HOME=/usr/local/hadoop
HADOOP_YARN_HOME=/usr/local/hadoop
HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
HADOOP_MAPRED_HOME=/usr/local/hadoop
So the script chooses this directory over the $HADOOP_HOME directory and subsequent subdirectories that are also called out in the script.  

The final step then was to edit the sqoop.properties file and ensure the mapreduce config was set to the correct directory:
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/etc/hadoop
Then the server started!

I'll leave this here in case anyone else runs across this...

                        
这篇关于启动Sqoop2服务器时无法找到hadoop配置类（1.99.7）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！