MySQL - Hive导入MySQL在Windows上运行，Hive在Cent OS上运行（Horton Sandbox） [英] Java - MySQL to Hive Import where MySQL Running on Windows and Hive Running on Cent OS (Horton Sandbox)

问题描述

在任何答复和评论之前。我尝试了几个在Stackoverflow中找到的选项，但以失败告终。以下是这些链接 -

• 使用sqoop-in-java-program问题/ 22487688 / how-to-import-table-from-mysql-to-hive-using-java>如何使用Java将表格从MySQL导入到Hive？


• 如何将SQL数据加载到Hortonworks？

我通过命令行在Horton Sandbox中尝试过它并成功完成了操作。

 
 sqoop import --connect jdbc：mysql：//192.168.56.101：3316 / database_name --username = user --password = pwd --table table_name --hive-import -m 1  -   - -schema默认
 

其中192.168.56.101为风对于Horton Sandbox 2.6而言，它的值为192.168.56.102。

现在我想从Java中完成同样的事情，即Java代码在别的地方运行，但不在horton沙箱中运行。 p>

• 如何布置HIVE_HOME和其他Sqoop参数，因为它们正在Sandbox中运行。 通过。它应该作为SqoopOptions或Sqoop.runTools字符串数组参数传递。这两个都失败。


• 我也很困惑当导入库（com.cloudera.sqoop和org.apache.sqoop）并得到这个

类型ImportTool中的run（com.cloudera.sqoop.SqoopOptions）方法不适用于带有这两行的参数
（org.apache.sqoop.SqoopOptions）选项参数添加在这两行之间）

  SqoopOptions options = new SqoopOptions（）; 
 int ret = new ImportTool（）。run（options）; 
  

如果我选择Cloudera方法不推荐使用，但如果我选择apace run方法不接受options参数

我从这个星期开始就感到惊讶。请帮助。

解决方案

是的，您可以通过ssh来完成。 Horton Sandbox附带预先安装的ssh支持。你可以在windows上通过ssh客户端执行sqoop命令。或者如果你想做它programaticaly（多数民众赞成我所做的java）你必须按照这一步。

1. 下载sshxcute java库： https://code.google.com/p/sshxcute/


2. 添加到包含以下java代码的java项目的构建路径中

---

  import net.neoremind.sshxcute.core.SSHExec; 
导入net.neoremind.sshxcute.core.ConnBean; 
导入net.neoremind.sshxcute.task.CustomTask; 
导入net.neoremind.sshxcute.task.impl.ExecCommand; 
 
 public class TestSSH {
 
 public static void main（String args []）throws Exception {
 
 //初始化一个ConnBean对象，参数列表是ip，用户名，密码
 
 ConnBean cb =新的ConnBean（192.168.56.102，root，hadoop）; 
 
 //将ConnBean实例作为SSHExec静态方法的参数getInstance（ConnBean）获取单例SSHExec实例
 SSHExec ssh = SSHExec.getInstance（cb）; 
 //连接服务器
 ssh.connect（）; 
 CustomTask sampleTask1 = new ExecCommand（echo $ SSH_CLIENT）; //打印您的客户IP通过它连接到Horton Sandbox上的ssh服务器
 System.out.println（ssh.exec（sampleTask1））; 
 CustomTask sampleTask2 = new ExecCommand（sqoop import --connect jdbc：mysql：//192.168.56.101：3316 / mysql_db_name --username = mysql_user --password = mysql_pwd --table mysql_table_name --hive-import -m 1  -  --schema默认）; 
 ssh.exec（sampleTask2）; 
 ssh.disconnect（）; 
} 
} 
  

---

Before any Answer and Comments. I tried several option I found in Stackoverflow but end with a failure. Following are those links -

• How can I execute Sqoop in Java?
• How to use Sqoop in Java Program?
• How to import table from MySQL to Hive using Java?
• How to load SQL data into the Hortonworks?

I tried it in Horton Sandbox through command line and succeded.

sqoop import --connect jdbc:mysql://192.168.56.101:3316/database_name --username=user --password=pwd --table table_name --hive-import -m 1 -- --schema default

Where 192.168.56.101 is for Windows and 192.168.56.102 for Horton Sandbox 2.6.

Now I want to do the same thing from Java where that java code run somewhere else but not in horton sandbox.

• How to loacate HIVE_HOME and other Sqoop parameters because that are running in Sandbox.
• parameters which I have to pass. It should be passes as SqoopOptions or Sqoop.runTools String Array Arguments. Both failing.
• I also get confused While import library (com.cloudera.sqoop and org.apache.sqoop) and get this

The method run(com.cloudera.sqoop.SqoopOptions) in the type ImportTool is not applicable for the arguments (org.apache.sqoop.SqoopOptions) with this two line (option parameter are added between this two lines)

 SqoopOptions options = new SqoopOptions();
 int ret = new ImportTool().run(options);

if I choose Cloudera method get deprecated but if I choose apace run method doesn't accept the options argument

I am strucked in this from weeks. Please Help.

解决方案

Yes you can do it via ssh. Horton Sandbox comes with ssh support pre installed. You can execute the sqoop command via ssh client on windows. Or if you want to do it programaticaly (thats what I have done in java) you have to follow this step.

1. Download sshxcute java library : https://code.google.com/p/sshxcute/
2. Add to the build path of your java project which contains the following java code

---

import net.neoremind.sshxcute.core.SSHExec;
import net.neoremind.sshxcute.core.ConnBean;
import net.neoremind.sshxcute.task.CustomTask;
import net.neoremind.sshxcute.task.impl.ExecCommand;

public class TestSSH {

public static void main(String args[]) throws Exception{

    // Initialize a ConnBean object, parameter list is ip, username, password

    ConnBean cb = new ConnBean("192.168.56.102", "root","hadoop");

    // Put the ConnBean instance as parameter for SSHExec static method getInstance(ConnBean) to retrieve a singleton SSHExec instance
    SSHExec ssh = SSHExec.getInstance(cb);          
    // Connect to server
    ssh.connect();
    CustomTask sampleTask1 = new ExecCommand("echo $SSH_CLIENT"); // Print Your Client IP By which you connected to ssh server on Horton Sandbox
    System.out.println(ssh.exec(sampleTask1));
    CustomTask sampleTask2 = new ExecCommand("sqoop import --connect jdbc:mysql://192.168.56.101:3316/mysql_db_name --username=mysql_user --password=mysql_pwd --table mysql_table_name --hive-import -m 1 -- --schema default");
    ssh.exec(sampleTask2);
    ssh.disconnect();   
}
}

---

这篇关于MySQL - Hive导入MySQL在Windows上运行，Hive在Cent OS上运行（Horton Sandbox）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！