使用Hiveserver2 Thrift Java客户端时，请求会挂起 [英] Requests hang when using Hiveserver2 Thrift Java client
问题描述
这是这个问题的后续问题，我问什么Hiveserver 2 thrift java客户端API是。如果你不需要任何更多的上下文，这个问题应该能够保持不变。
无法找到关于如何使用hiverserver2 thrift api的任何文档，我把它放在一起。我能找到的最佳参考是 Apache JDBC实现。
TSocket传输=新TSocket（hive.example.com，10002）;
transport.setTimeout（999999999）;
TBinaryProtocol protocol = new TBinaryProtocol（transport）;
TCLIService.Client client = new TCLIService.Client（protocol）;
transport.open（）;
TOpenSessionReq openReq = new TOpenSessionReq（）;
TOpenSessionResp openResp = client.OpenSession（openReq）;
TSessionHandle sessHandle = openResp.getSessionHandle（）;
TExecuteStatementReq execReq = new TExecuteStatementReq（sessHandle，SHOW TABLES）;
TExecuteStatementResp execResp = client.ExecuteStatement（execReq）;
TOperationHandle stmtHandle = execResp.getOperationHandle（）;
TFetchResultsReq fetchReq = new TFetchResultsReq（stmtHandle，TFetchOrientation.FETCH_FIRST，1）;
TFetchResultsResp resultsResp = client.FetchResults（fetchReq）;
TRowSet resultsSet = resultsResp.getResults（）;
列表< TRow> resultRows = resultsSet.getRows（）; $（b $ b $ for（TRow resultRow：resultRows）{
resultRow.toString（）;
}
TCloseOperationReq closeReq = new TCloseOperationReq（）;
closeReq.setOperationHandle（stmtHandle）;
client.CloseOperation（closeReq）;
TCloseSessionReq closeConnectionReq = new TCloseSessionReq（sessHandle）;
client.CloseSession（closeConnectionReq）;
transport.close（）;
我使用 $ b $创建的Hiverserver2实例运行此代码b
export HIVE_SERVER2_THRIFT_PORT = 10002; hive --service hiveserver2
在调试时，我从来没有超过这一行
TOpenSessionResp openResp = client.OpenSession（openReq）;
客户端简单地挂起，直到达到超时并且服务器不向stdout写入任何内容或日志。使用Wireshark，我可以看到OpenSession（）的TCP段被发送并确认。一旦我杀死客户端或达到超时，服务器给我以下内容：
13/03/14 11:15 ：33错误server.TThreadPoolServer：处理消息期间发生错误。
java.lang.RuntimeException：org.apache.thrift.transport.TTransportException：java.net.SocketException：连接重置
at org.apache.thrift.transport.TSaslServerTransport $ Factory.getTransport（TSaslServerTransport.java ：219）
at org.apache.thrift.server.TThreadPoolServer $ WorkerProcess.run（TThreadPoolServer.java:189）
at java.util.concurrent.ThreadPoolExecutor $ Worker.runTask（ThreadPoolExecutor.java:886 ）
在java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:908）$ b $在java.lang.Thread.run（Thread.java:662）
导致： org.apache.thrift.transport.TTransportException：java.net.SocketException：连接重置
在org.apache.thrift.transport.TIOStreamTransport.read（TIOStreamTransport.java:129）
在org.apache。 thrift.transport.TTransport.readAll（TTransport.java:84）
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage（TSaslTransport.java:182）
at org.apache.thrift.transport。 TSAS lServerTransport.handleSaslStartMessage（TSaslServerTransport.java:125）
at org.apache.thrift.transport.TSaslTransport.open（TSaslTransport.java:253）
at org.apache.thrift.transport.TSaslServerTransport.open（ TSaslServerTransport.java:41）
at org.apache.thrift.transport.TSaslServerTransport $ Factory.getTransport（TSaslServerTransport.java:216）
... 4 more
导致：java.net .SocketException：连接在java.net.SocketInputStream.read（SocketInputStream.java:168）上重置
在java.io.BufferedInputStream.read1上的
（BufferedInputStream.java:256）在java上的
。 io.BufferedInputStream.read（BufferedInputStream.java:317）
at org.apache.thrift.transport.TIOStreamTransport.read（TIOStreamTransport.java:127）
... 10 more
我发现有趣的是，当我错误地尝试使用hiveserver（1）客户端时，反对hiveserver2，这表明迄今为止s hiverserver2而言，我的客户正在发送垃圾。
我看到三种可能性，我可能会出错。
1）我使用客户端API是错误的。我看到在JDBC实现中有一些事情正在进行，我没有在我的示例代码中使用身份验证和连接参数。我玩过这个游戏，但是我在黑暗中拍摄，并没有再进一步。
2）我有一些设置步骤错了。我无法在hive-servive-0.10.0 jar中找到TCLIService，但是我可以在Hortonworks在HDP 1.2中发布的hive-servive-0.10.0.21 jar中找到它，所以也许可以用它来挖掘它。揭示了这个问题。或者，也许有一些我需要配置服务器端，这解释了为什么我可以连接到配置单元使用ODBC，但不是与我的节俭客户端。
3）可能是在这一点是不可能写入hiveserver2客户端API的。基于缺乏文档和互联网上成功案例的明显缺乏，这似乎是合理的，但JDBC似乎做到了。我发现这是最不可能的选择。
即使您不知道修补程序，知道该修补程序是否低于1,2或3，将有助于缩小搜索范围。
不知道您是否仍遇到此问题，但由于我遇到同样的问题并解决了它（也许绕过是更准确的描述），我会发布一个解决方案，以防万一别人需要它。
这是因为节俭服务器期望通过SASL进行身份验证当你打开你的交通连接。 Hive Server 2默认使用SASL--不幸的是，PHP缺少一个版本的TSaslClientTransport（用作另一个TTransport对象的包装），它在您打开传输连接时处理SASL协商。
现在最简单的解决方案是在您的hive-site.xml中设置以下属性
< property> ;<名称>&hive.server2.authentication LT; /名称><值GT; NOSASL< /值GT;< /性>
This is a follow up question to this question where I ask what the Hiveserver 2 thrift java client API is. This question should be able to stand along without that background if you don't need any more context.
Unable to find any documentation on how to use the hiverserver2 thrift api, I put this together. The best reference I could find was the Apache JDBC implementation.
TSocket transport = new TSocket("hive.example.com", 10002);
transport.setTimeout(999999999);
TBinaryProtocol protocol = new TBinaryProtocol(transport);
TCLIService.Client client = new TCLIService.Client(protocol);
transport.open();
TOpenSessionReq openReq = new TOpenSessionReq();
TOpenSessionResp openResp = client.OpenSession(openReq);
TSessionHandle sessHandle = openResp.getSessionHandle();
TExecuteStatementReq execReq = new TExecuteStatementReq(sessHandle, "SHOW TABLES");
TExecuteStatementResp execResp = client.ExecuteStatement(execReq);
TOperationHandle stmtHandle = execResp.getOperationHandle();
TFetchResultsReq fetchReq = new TFetchResultsReq(stmtHandle, TFetchOrientation.FETCH_FIRST, 1);
TFetchResultsResp resultsResp = client.FetchResults(fetchReq);
TRowSet resultsSet = resultsResp.getResults();
List<TRow> resultRows = resultsSet.getRows();
for(TRow resultRow : resultRows){
resultRow.toString();
}
TCloseOperationReq closeReq = new TCloseOperationReq();
closeReq.setOperationHandle(stmtHandle);
client.CloseOperation(closeReq);
TCloseSessionReq closeConnectionReq = new TCloseSessionReq(sessHandle);
client.CloseSession(closeConnectionReq);
transport.close();
I run this code against a Hiverserver2 instance created with
export HIVE_SERVER2_THRIFT_PORT=10002;hive --service hiveserver2
When debugging, I never get past the line
TOpenSessionResp openResp = client.OpenSession(openReq);
The client simply hangs until the timeout is reached and the server doesn't write anything to stdout or the logs. Using Wireshark, I can see the TCP segment for OpenSession() is sent and ACK'd. Once I kill the client or the timeout is reached, the server gives me the following:
13/03/14 11:15:33 ERROR server.TThreadPoolServer: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 4 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 10 more
I find it interesting that this is the exact same error I was receiving when I was mistakenly attempting to use a hiveserver (1) client against hiveserver2, which suggests that as far as hiverserver2 is concerned, my client is sending it garbage.
I see three possibilities for where I might be going wrong.
1) My use of the client API is wrong. I saw that in the JDBC implementation there was some stuff going on with authentication and connection parameters which I'm not using in my example code. I played around with that, but I was shooting in the dark and didn't get any further.
2) I got some setup step wrong. I wasn't able to find TCLIService in the hive-servive-0.10.0 jar, but I was able to find it in the hive-servive-0.10.0.21 jar released by Hortonworks in HDP 1.2, so maybe digging around with that will reveal the issue. Or maybe there is something I need configure server side which explains why I can connect to hive using the ODBC but not with my thrift client.
3) It could be that at this point it is impossible to write against the hiveserver2 client api. This is plausible based on the lack of documentation and the apparent lack of successful examples on the internet, but the JDBC seems to do it. I find this the most unlikely option.
Even if you don't know a fix, knowing if the fix falls under 1, 2, or 3 would help narrow my search.
Not sure if you're still experiencing this issue, but since i've confronted the same problem and resolved it (perhaps bypassed is more accurate description), i'll post a solution here just in case someone else needs it.
This is because the thrift server is expecting to authenticate via SASL when you open your transport connection. Hive Server 2 defaults to using SASL - unfortunately, PHP lacks a version of TSaslClientTransport (which is used as a wrapper around another TTransport object) which handles the SASL negotiation when you open your transport connection.
The easiest solution for now is to set the following property in your hive-site.xml
<property><name>hive.server2.authentication</name><value>NOSASL</value></property>
这篇关于使用Hiveserver2 Thrift Java客户端时，请求会挂起的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
