我如何在Java中执行Sqoop? [英] How can I execute Sqoop in Java?
问题描述
我创建了一个新的Java项目,然后添加了Library Sqoop和Hadoop。 (这些库是hadoop-core-1.1.1.jar,sqoop-1.4.2.jar等....)。
然后我尝试下面代码:
public class MySqoopDriver {
public static void main(String [] args){
String [] str = {export,--connect,jdbc:mysql:// localhost / mytestdb,--hadoop-home,
/home/yoonhok/development/hadoop-1.1 .1,--table,tbl_1,--export-dir,hdfs:// localhost:9000 / user / hive / warehouse / tbl_1,
--username, yoonhok,--password,1234};
Sqoop.runTool(str);
$ b $ p
$ b 参数是正确的,因为当我尝试在终端,它运作良好。
但它没有奏效。错误消息是:
13/02/17 16:23:07 WARN tool.BaseSqoopTool:在命令中设置密码 - 线是不安全的。考虑使用-P来代替。
13/02/17 16:23:07信息manager.MySQLManager:准备使用MySQL流结果集。
13/02/17 16:23:07 INFO tool.CodeGenTool:开始代码生成
13/02/17 16:23:07 INFO manager.SqlManager:执行SQL语句:SELECT t。* FROM `tbl_1` as t LIMIT 1
13/02/17 16:23:07 INFO manager.SqlManager:执行SQL语句:SELECT t。* FROM`tbl_1` AS t LIMIT 1
13/02 / 17 16:23:07 INFO orm.CompilationManager:HADOOP_HOME是/home/yoonhok/development/hadoop-1.1.1
注意:/tmp/sqoop-yoonhok/compile/86a3cab62184ad50a3ae11e7cb0e4f4d/tbl_1.java使用或覆盖弃用API。
注意:使用-Xlint:deprecation重新编译以获取详细信息。
13/02/17 16:23:08 INFO orm.CompilationManager:编写jar文件:/tmp/sqoop-yoonhok/compile/86a3cab62184ad50a3ae11e7cb0e4f4d/tbl_1.jar
13/02/17 16:23: 08信息mapreduce.ExportJobBase:开始导出tbl_1
13/02/17 16:23:09 WARN util.NativeCodeLoader:无法为您的平台加载native-hadoop库......使用内置java类(适用时)
13/02/17 16:23:09 INFO input.FileInputFormat:要输入的总输入路径:1
13/02/17 16:23:09 INFO input.FileInputFormat:要输入的总输入路径: 1
13/02/17 16:23:09信息mapred.JobClient:清理临时区域文件:/tmp/hadoop-yoonhok/mapred/staging/yoonhok1526809600/.staging/job_local_0001
13 / 02/17 16:23:09错误security.UserGroupInformation:PriviledgedActionException as:yoonhok cause:java.io.FileNotFoundException:File / user / hive / warehouse / tbl_1 / 000000_0不存在。
13/02/17 16:23:09 ERROR tool.ExportTool:遇到IOException运行导出作业:java.io.FileNotFoundException:File / user / hive / warehouse / tbl_1 / 000000_0不存在。
当我检查HDFS时,文件存在:
hadoop fs -ls / user / hive / warehouse / tbl_1
找到1项
-rw-r - r-- 1 yoonhok supergroup 240 2013 -02-16 18:56 / user / hive / warehouse / tbl_1 / 000000_0
如何执行在我的Java程序中使用Sqoop?
我尝试了Processbuilder和Process,但我不想使用它们。
<我真的很想使用Sqoop API,但我听说它还不存在。
我阅读,但它不适合我。
解决方案有一个技巧对我来说非常简单。通过ssh你可以直接执行Sqoop命令。只要你必须使用的是一个SSH Java库
你必须遵循这一步。
下载sshxcute java库: https://code.google.com/p/sshxcute/
并将其添加到包含以下java代码的java项目的构建路径中。
import net.neoremind.sshxcute.core。 SSHExec;
导入net.neoremind.sshxcute.core.ConnBean;
导入net.neoremind.sshxcute.task.CustomTask;
导入net.neoremind.sshxcute.task.impl.ExecCommand;
public class TestSSH {
public static void main(String args [])throws Exception {
//初始化一个ConnBean对象,参数列表是ip,用户名,密码
ConnBean cb =新的ConnBean(192.168.56.102,root,hadoop);
//将ConnBean实例作为SSHExec静态方法的参数getInstance(ConnBean)获取单例SSHExec实例
SSHExec ssh = SSHExec.getInstance(cb);
//连接服务器
ssh.connect();
CustomTask sampleTask1 = new ExecCommand(echo $ SSH_CLIENT); //打印您的客户IP通过它连接到Horton Sandbox上的ssh服务器
System.out.println(ssh.exec(sampleTask1));
CustomTask sampleTask2 = new ExecCommand(sqoop import --connect jdbc:mysql://192.168.56.101:3316 / mysql_db_name --username = mysql_user --password = mysql_pwd --table mysql_table_name --hive-import -m 1 - --schema默认);
ssh.exec(sampleTask2);
ssh.disconnect();
}
}
I created a new Java project, then I added Library Sqoop and Hadoop. (The Libraries are "hadoop-core-1.1.1.jar, sqoop-1.4.2.jar, etc.... ".)
Then I tried below code:
public class MySqoopDriver {
public static void main(String[] args) {
String[] str = { "export", "--connect", "jdbc:mysql://localhost/mytestdb", "--hadoop-home",
"/home/yoonhok/development/hadoop-1.1.1", "--table", "tbl_1", "--export-dir", "hdfs://localhost:9000/user/hive/warehouse/tbl_1",
"--username", "yoonhok", "--password", "1234"};
Sqoop.runTool(str);
}
}
The parameters are right, since when I try in terminal, it works well.
But it didn't work. The error message is:
13/02/17 16:23:07 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
13/02/17 16:23:07 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
13/02/17 16:23:07 INFO tool.CodeGenTool: Beginning code generation
13/02/17 16:23:07 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_1` AS t LIMIT 1
13/02/17 16:23:07 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_1` AS t LIMIT 1
13/02/17 16:23:07 INFO orm.CompilationManager: HADOOP_HOME is /home/yoonhok/development/hadoop-1.1.1
Note: /tmp/sqoop-yoonhok/compile/86a3cab62184ad50a3ae11e7cb0e4f4d/tbl_1.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/02/17 16:23:08 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-yoonhok/compile/86a3cab62184ad50a3ae11e7cb0e4f4d/tbl_1.jar
13/02/17 16:23:08 INFO mapreduce.ExportJobBase: Beginning export of tbl_1
13/02/17 16:23:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/02/17 16:23:09 INFO input.FileInputFormat: Total input paths to process : 1
13/02/17 16:23:09 INFO input.FileInputFormat: Total input paths to process : 1
13/02/17 16:23:09 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-yoonhok/mapred/staging/yoonhok1526809600/.staging/job_local_0001
13/02/17 16:23:09 ERROR security.UserGroupInformation: PriviledgedActionException as:yoonhok cause:java.io.FileNotFoundException: File /user/hive/warehouse/tbl_1/000000_0 does not exist.
13/02/17 16:23:09 ERROR tool.ExportTool: Encountered IOException running export job: java.io.FileNotFoundException: File /user/hive/warehouse/tbl_1/000000_0 does not exist.
When I check the HDFS, the file exists:
hadoop fs -ls /user/hive/warehouse/tbl_1
Found 1 items
-rw-r--r-- 1 yoonhok supergroup 240 2013-02-16 18:56 /user/hive/warehouse/tbl_1/000000_0
How can I execute Sqoop in my Java program?
I tried Processbuilder and Process, but I don't want to use them.
I really want to use the Sqoop API, but I heard that it doesn't exist yet.
I read this question but it doesn't work for me.
解决方案 There is a trick which worked out for me pretty easy. Via ssh you can execute Sqoop command directly. Just you have to use is a SSH Java Library
You have to follow this step.
Download sshxcute java library : https://code.google.com/p/sshxcute/
and Add it to the build path of your java project which contains the following java code
import net.neoremind.sshxcute.core.SSHExec;
import net.neoremind.sshxcute.core.ConnBean;
import net.neoremind.sshxcute.task.CustomTask;
import net.neoremind.sshxcute.task.impl.ExecCommand;
public class TestSSH {
public static void main(String args[]) throws Exception{
// Initialize a ConnBean object, parameter list is ip, username, password
ConnBean cb = new ConnBean("192.168.56.102", "root","hadoop");
// Put the ConnBean instance as parameter for SSHExec static method getInstance(ConnBean) to retrieve a singleton SSHExec instance
SSHExec ssh = SSHExec.getInstance(cb);
// Connect to server
ssh.connect();
CustomTask sampleTask1 = new ExecCommand("echo $SSH_CLIENT"); // Print Your Client IP By which you connected to ssh server on Horton Sandbox
System.out.println(ssh.exec(sampleTask1));
CustomTask sampleTask2 = new ExecCommand("sqoop import --connect jdbc:mysql://192.168.56.101:3316/mysql_db_name --username=mysql_user --password=mysql_pwd --table mysql_table_name --hive-import -m 1 -- --schema default");
ssh.exec(sampleTask2);
ssh.disconnect();
}
}
这篇关于我如何在Java中执行Sqoop?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!