如何修复java.lang.IllegalArgumentException:无法识别的Hadoop主版本号:3.1.0? [英] How can I fix java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0?

问题描述

我在查询中得到一个java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0 exception.这是查询:

I get a java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0 exception in my query. Here's the query:

WITH 
t1 as 
(select * from browserdata join citydata on cityid=id),

t2 as 
(select uap.device as device, uap.os as os, uap.browser as browser, name as cityname 
from t1 
lateral view ParseUserAgentUDTF(UserAgent) uap as device, os, browser),

t3 as
(select t2.cityname as cityname, t2.device as device, t2.browser as browser, t2.os as os, count(*) as count from t2 group by t2.cityname, t2.os, t2.device, t2.browser)

select cityname, maximum,  device, os, browser
 from
     (select cityname, device, browser, os, 
             max(count) over(partition by cityname)                         as maximum,
             dense_rank() over (partition by cityname order by count desc ) as rnk      
      from t3
     ) s  where rnk =1 
;

这是我的容器中的日志:

And here's the log from my container:

Log Type: stdout

Log Upload Time: Mon Dec 24 16:21:37 +0000 2018

Log Length: 5529

Showing 4096 bytes of 5529 total. Click here for the full log.

.8.0_171]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682) [udf.jar:?]
    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) [tez-runtime-internals-0.7.0.2.6.5.0-292.jar:0.7.0.2.6.5.0-292]
    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) [tez-runtime-internals-0.7.0.2.6.5.0-292.jar:0.7.0.2.6.5.0-292]
    at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) [tez-common-0.7.0.2.6.5.0-292.jar:0.7.0.2.6.5.0-292]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.1.0
    at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:178) ~[hive-exec-1.2.1000.2.6.5.0-292-d249a9484f801bbb96f01e7bbd357a58127aaca3e59c783a90c062bf99c9310d.jar:1.2.1000.2.6.5.0-292]
    at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:143) ~[hive-exec-1.2.1000.2.6.5.0-292-d249a9484f801bbb96f01e7bbd357a58127aaca3e59c783a90c062bf99c9310d.jar:1.2.1000.2.6.5.0-292]
    at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:102) ~[hive-exec-1.2.1000.2.6.5.0-292-d249a9484f801bbb96f01e7bbd357a58127aaca3e59c783a90c062bf99c9310d.jar:1.2.1000.2.6.5.0-292]
    at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:452) ~[hive-exec-1.2.1000.2.6.5.0-292-d249a9484f801bbb96f01e7bbd357a58127aaca3e59c783a90c062bf99c9310d.jar:1.2.1000.2.6.5.0-292]
    ... 16 more

我无法理解问题所在，所有内容都可以在mapreduce上使用，但不想在tez上使用.

I haven't been able to understand what the problem is, everything works on mapreduce but doesn't want to work on tez.

我为此查询使用用户定义的函数来分析其中一列中的用户代理字符串.

I use a user defined function for this query to parse the user agent string in one of the columns.

推荐答案

您正在运行适用于HDP 2.6.5的Hortonworks-sandbox，它具有Hadoop 2.7.x和Hive 1.2，而不是Hadoop/Hive 3

You're running Hortonworks-sandbox for HDP 2.6.5, which has Hadoop 2.7.x and Hive 1.2, not Hadoop/Hive 3

这篇关于如何修复java.lang.IllegalArgumentException:无法识别的Hadoop主版本号:3.1.0?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！