Hive Transactions正在崩溃 [英] Hive Transactions are crashing

问题描述

好的，我一直在处理这个问题几天，这让我感到非常紧张。我需要在事务中使用Hive数据库来执行'update'和'delete'操作。

我在我的机器上以伪分布模式安装了Hadoop和Hive。我已按照此教程进行安装。我使用的是Java 1.8.0_31，Hadoop 2.6.0，Hive 1.0.0，并且我还修改了一些细节，但这些细节不应该相关。

<现在，为了启动我的环境（例如重启后），我运行以下命令：

  start-dfs。 sh 
 start-yarn.sh 
 java -jar /usr/local/derby/lib/derbyrun.jar服务器启动& 
 hive 
  

一切似乎都正常。尽管本教程没有提到启动德比，但如果我不启动它，Metastore不可用（这看起来合乎逻辑），并且配置单元不会启动。

从这里，我可以创建表格，显示表格，连接我的JDBC客户端等等，一切都很好。现在，我需要启用交易。遵循此链接并在此< href =http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_dataintegration/content/ch_using-hive-transactions.html =nofollow noreferrer>链接我得到到以下命令：

  hive --hiveconf hive.root.logger = info，console 
 --hiveconf hive .txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager 
 --hiveconf hive.compactor.initiator.on = true 
 --hiveconf hive.compactor.worker.threads = 1 
 --hiveconf hive.txn.driver = jdbc：derby：// localhost：1527 / metastore_db; create = true 
  

旁注：我正在更改命令而不是hive-site.xml，只是因为在尝试执行某些操作时可以更轻松地更改命令，并且哪些操作不起作用，而不是反复更改XML文件。 p>

我也尝试将驱动程序url更改为 jdbc：derby：// localhost：1527 / metastore_db; create = true; user = APP; password = mine 以防万一需要时，但没有改变。当我发出一个命令时（比如 show tables ），我得到一个错误：

  15/03/04 23:26:17 [main]：错误metastore.RetryingHMSHandler：
 MetaException（消息：无法从事务数据库中选择，
 java.sql.SQLSyntaxErrorException：Table / View 'TXNS'不存在
  

根据这个和其中一个以前的链接，它好像是 hive.in.test 属性必须设置为 true ，所以我的启动命令变为：

  hive --hiveconf hive.root.logger = info，console 
 --hiveconf hive.in.test = true 
 --hiveconf hive。 txn.manager = org.apache.hadoop.hive.ql.lockmgr.D bTxnManager 
 --hiveconf hive.compactor.initiator.on = true 
 --hiveconf hive.compactor.worker.threads = 1 
 --hiveconf hive.txn.driver = jdbc：derby： //本地主机：1527 / metastore_db;创建= TRUE; 
  

使用此命令，我收到一个新错误：

  ERROR metastore.RetryingHMSHandler：java.lang.NullPointerException $ b $在org.apache.hadoop.hive.metastore.txn.TxnHandler.checkQFileTestHack（TxnHandler.java:1146） 
  

这个错误在任何地方都不存在，我觉得我是互联网上唯一的人用它。无论如何，因为我找不到任何解决方案，我深入到源代码中：

pre $ private $ checkQFileTestHack $ b布尔值hackOn = HiveConf.getBoolVar（conf，HiveConf.ConfVars.HIVE_IN_TEST）||
HiveConf.getBoolVar（conf，HiveConf.ConfVars.HIVE_IN_TEZ_TEST）;

LOG.info（if if）;
if（hackOn）{
LOG.info（对事务管理器的黑箱入库值）;
//在derby Metastore中设置事务/锁定数据库
TxnDbUtil.setConfValues（conf）;
尝试{
TxnDbUtil.prepDb（）;
} catch（Exception e）{
//我们可能已经创建了表格，因此不需要重做它。
if（！e.getMessage（）。contains（already exists））{
throw new RuntimeException（无法为+
设置事务数据库测试：+ e .getMessage（））;






$ p行1146是 if（！e.getMessage（）。contains（already exists）） line，这似乎没有太大意义，除非e是一个空，这很奇怪。无论如何，我想我可以通过添加更多日志消息，构建项目并替换原来的metastore jar（这是这个TxnHandler类所在的地方）来修改这个问题。为此，我下载了源代码并遵循此建立它。我尝试了maven2，但它不工作，因为一些插件只能用于maven3，所以我从这里并构建了该项目。

如果我使用 mvn clean install -Phadoop-2，dist 命令构建它，不仅需要永远，但在测试阶段失败。因为它在Metastore上没有失败（在Metastore上，它跳过了1次测试，我不确定是否应该发生），我想我可以在没有测试的情况下建立它。所以，我们可以这样做：

  mvn clean install -DskipTests -Phadoop-2，dist 
 rm / usr / local / hive / lib / hive-metastore-1.0.0.jar 
 cp packaging / target / apache-hive-1.0.0-bin / apache-hive-1.0.0-bin / lib / hive-metastore- 1.0.0.jar / usr / local / hive / lib / 
  

旁注：为了时间的利益，我也尝试过 -pl metastore -am 参数，但maven说Metastore已经建好，lib文件夹中的jar不会改变，所以我猜测我做错了什么。

无论如何，这应该建立我的修改过的jar，替换配置单元中的一个，并且当我再次启动配置单元时，它应该加载我的。但是，即使在我更改代码后，错误仍显示相同，我的新日志信息未注册，即使错误行保持不变。这就像我在新的罐子里没有改变任何东西。它奇怪的，我知道maven编译我的代码，因为它承认编译错误，我可以看到在jar的属性，这是一个新的文件，所以为什么不我的其他变化显示出来了吗？ Hive能够识别出我删除原始jar的时候，但是当我用我的修改版本替换它时，它就像我没有改变任何东西。

无论如何，正如你所看到的，我已经有很多麻烦，我试图修复其中的大部分。但现在我坚持在这一个，没有能够使用一个该死的删除命令，因为我不能启用交易。任何人都可以将我指向正确的方向吗？ Tyvm！

...并且很抱歉发布了很长的帖子。

解决方案

<我跟着Srinivas'建议，错误消失。我不再需要将hive.in.test属性设置为true，并且一切正常。

我仍然不知道为什么更改源文件不会影响其余的配置单元程序，但我有交易工作。

编辑：如果链接关闭，这里是一个报价：

在提取Hive版本后，您必须创建Hive元存储

  sudo apt- get install mysql-server 
 sudo服务mysql start $ b $ sudo apt-get install libmysql -java 
 ln -s /usr/share/java/libmysql-java.jar / usr / lib / hive /lib/libmysql-java.jar 
 sudo chkconfig mysql 
 $ b mysql -u root -p 
输入密码：
 mysql> CREATE DATABASE metastore; 
 mysql>使用metastore; 
 mysql> SOURCE /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.12.0.mysql.sql; 
 
 mysql> CREATE USER'hive'@'metastorehost'IDENTIFIED BY'mypassword'; 
 ... 
 mysql> REVOKE ALL PRIVILEGES，授予选项'hive'@'metastorehost'; 
 mysql> GRANT SELECT，INSERT，UPDATE，DELETE，LOCK TABLES，EXECUTE ON metastore。* TO'hive'@'metastorehost'; 
 mysql> FLUSH PRIVILEGES; 
 mysql>放弃; 
  

然后在hive-site.xml中，您需要设置新的参数，如

  javax.jdo.option.ConnectionURL  -  jdbc：mysql：// myhost / metastore 
 javax.jdo.option.ConnectionDriverName  -  com.mysql .jdbc.Driver 
 javax.jdo.option.ConnectionUserName  -  hive 
 javax.jdo.option.ConnectionPassword  -  mypassword 
 datanucleus.autoCreateSchema  -  false 
 datanucleus.fixedDatastore  -  true 
 datanucleus.autoStartMechanism  -  SchemaTable 
 hive.metastore.uris  -  thrift：//< nnnn>：9083 
 
 hive.support.concurrency  -  true 
 hive.enforce .bucketing  -  true 
 hive.exec.dynamic.partition.mode  - 非限制
 hive.txn.manager  -  org.apache.hadoop.hive.ql.lockmgr.DbTxnManager 
 hive.compactor。 initiator.on  -  true 
 hive.compactor.worker.threads  -  1 
  

然后重新启动Hive服务器和Metastore。现在创建一个普通表
和一个带有orc格式的外部表，并从正常到orc
表加载。现在您可以更新和删除记录。

Ok, I've been dealing with this problem for a couple of days and it's driving me nuts. I need to use the Hive database with transactions to perform 'update' and 'delete' operations.

I have installed Hadoop and Hive on my machine in pseudo-distributed mode. I have followed this tutorial for the installation. I'm using Java 1.8.0_31, Hadoop 2.6.0, Hive 1.0.0 and there were also a couple of details I changed, but these shouldn't be relevant.

Now, to start my environment (after a reboot, for example), i run the following:

start-dfs.sh
start-yarn.sh
java -jar /usr/local/derby/lib/derbyrun.jar server start &
hive

And everything seems to work fine. Although the tutorial doesn't mention starting derby, if i don't start it, the metastore isn't available (which seems logical) and hive doesn't start.

From here, i can create tables, show tables, connect with my JDBC client, etc etc, everything works great. Now, i need to enable transactions. Following this link and this link i get to the following command:

hive --hiveconf hive.root.logger=info,console 
    --hiveconf hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager 
    --hiveconf hive.compactor.initiator.on=true 
    --hiveconf hive.compactor.worker.threads=1 
    --hiveconf hive.txn.driver=jdbc:derby://localhost:1527/metastore_db;create=true

Sidenote: I'm changing the command and not hive-site.xml just because it's easier to change between commands when trying what works and what doesn't work instead of repeatedly changing the XML file.

I have also tried changing the driver url to jdbc:derby://localhost:1527/metastore_db;create=true;user=APP;password=mine just in case it was needed, but there's no change. When i issue a command (like show tables), i get an error:

15/03/04 23:26:17 [main]: ERROR metastore.RetryingHMSHandler: 
    MetaException(message:Unable to select from transaction database, 
    java.sql.SQLSyntaxErrorException: Table/View 'TXNS' does not exist.

According to this and one of the previous links, it seems like the hive.in.test property must be set to true. So, my launch command becomes:

hive --hiveconf hive.root.logger=info,console 
    --hiveconf hive.in.test=true 
    --hiveconf hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager 
    --hiveconf hive.compactor.initiator.on=true 
    --hiveconf hive.compactor.worker.threads=1 
    --hiveconf hive.txn.driver=jdbc:derby://localhost:1527/metastore_db;create=true;

With this command, I get a new error:

ERROR metastore.RetryingHMSHandler: java.lang.NullPointerException
    at org.apache.hadoop.hive.metastore.txn.TxnHandler.checkQFileTestHack(TxnHandler.java:1146)

And this error doesn't exist anywhere, i feel like i'm the only person on the internet with it. Anyway, because i couldn't find any solution, I dug into the source code:

private void checkQFileTestHack() {
  boolean hackOn = HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST) ||
    HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEZ_TEST);

  LOG.info("Before if");
  if (hackOn) {
      LOG.info("Hacking in canned values for transaction manager");
      // Set up the transaction/locking db in the derby metastore
      TxnDbUtil.setConfValues(conf);
      try {
          TxnDbUtil.prepDb();
      } catch (Exception e) {
          // We may have already created the tables and thus don't need to redo it.
          if (!e.getMessage().contains("already exists")) {
              throw new RuntimeException("Unable to set up transaction database for" +
                " testing: " + e.getMessage());
          }
      }
  }
}

Line 1146 is the if (!e.getMessage().contains("already exists")) line, which doesn't seem to make much sense, unless "e" is a null, which is strange. Anyway, I thought i could debug this further by adding a few more logging messages, building the project and replacing the original metastore jar (which is where this TxnHandler class is) which my modified one. For that, i downloaded the source code and followed this to build it. I tried maven2 and it didn't work, because some plug-in only worked with maven3, so I got maven3 from here and built the project.

If i build it with the mvn clean install -Phadoop-2,dist command, not only does it take forever, but it fails during the test phase. Because it doesn't fail on the metastore (on the metastore, it skips 1 test, i'm not sure that's supposed to happen), i thought i could just build it without testing. So, we get to this:

mvn clean install -DskipTests -Phadoop-2,dist
rm /usr/local/hive/lib/hive-metastore-1.0.0.jar
cp packaging/target/apache-hive-1.0.0-bin/apache-hive-1.0.0-bin/lib/hive-metastore-1.0.0.jar /usr/local/hive/lib/

Sidenote: in the interest of time, i also tried the -pl metastore -am arguments, but while maven says that metastore has been built, the jar in the lib folder does not change, so I'm guessing I'm doing something wrong.

Anyway, this should build my modified jar, replace the one in hive and, when i start hive again, it should load mine. However, even after i change the code, the error still shows the same, my new logging info isn't registered, even the error line remains the same. It's like i changed nothing in my new jar.

Its strange, i know maven is compiling my code because it recognizes compile errors and i can see on the jar properties that it's a new file, so why don't the rest of my changes show up? Hive recognizes when I delete the original jar, but when I replace it with my modified version, its like I changed nothing.

Anyway, as you can see, i've had many troubles and i've tried to fix most of them. But now im stuck in this one, without being able to use a damn "delete" command because i cant enable transactions. Can anyone point me in the right direction? Tyvm!

... and sorry for the long post.

解决方案

I followed Srinivas' advice and the error disappeared. I no longer need the "hive.in.test" property set to "true" and everything works fine.

I still dont know why changing the source wouldnt affect the rest of the hive program, but I have transactions working.

Edit: in case the link goes down, here's a quote:

After extracting Hive version, you have to create Hive meta store

sudo apt-get install mysql-server
sudo service mysql start
sudo apt-get install libmysql-java
ln -s /usr/share/java/libmysql-java.jar /usr/lib/hive/lib/libmysql-java.jar
sudo chkconfig mysql on

mysql -u root -p
Enter password:
mysql> CREATE DATABASE metastore;
mysql> USE metastore;
mysql> SOURCE /usr/lib/hive/scripts/metastore/upgrade/mysql/hive-schema-0.12.0.mysql.sql;

mysql> CREATE USER 'hive'@'metastorehost' IDENTIFIED BY 'mypassword';
...
mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'hive'@'metastorehost';
mysql> GRANT SELECT,INSERT,UPDATE,DELETE,LOCK TABLES,EXECUTE ON metastore.* TO 'hive'@'metastorehost';
mysql> FLUSH PRIVILEGES;
mysql> quit;

Then in hive-site.xml, you need set the new parameters like

javax.jdo.option.ConnectionURL - jdbc:mysql://myhost/metastore
javax.jdo.option.ConnectionDriverName - com.mysql.jdbc.Driver
javax.jdo.option.ConnectionUserName - hive
javax.jdo.option.ConnectionPassword - mypassword
datanucleus.autoCreateSchema - false
datanucleus.fixedDatastore - true
datanucleus.autoStartMechanism - SchemaTable
hive.metastore.uris - thrift://<n.n.n.n>:9083

hive.support.concurrency – true  
hive.enforce.bucketing – true 
hive.exec.dynamic.partition.mode – nonstrict  
hive.txn.manager – org.apache.hadoop.hive.ql.lockmgr.DbTxnManager 
hive.compactor.initiator.on – true  
hive.compactor.worker.threads – 1

Then restart Hive-server and Metastore. Now create one normal table and one external table with orc format and load from normal to orc table. Now you can update and delete records.

这篇关于Hive Transactions正在崩溃的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！