找不到:org.apache.hadoop.security.authentication.util.KerberosUtil [英] Not found :org.apache.hadoop.security.authentication.util.KerberosUtil

问题描述

我正在集群中运行Storm jar，在其中配置了hadoop，kafka，storm集群

I am running storm jar in a cluster ,where I configured hadoop,kafka,storm cluster

当我在本地模式下运行jar时，它运行良好，当我在Storm集群上运行它时，我在Storm UI中发现了各自的错误:

when I run the jar in local mode it works fine ,when I run it on storm cluster, I am finding respective error in Storm UI:

java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosTicket(Ljavax/security/auth/Subject;)Z at 
org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:666) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:861) at 
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)

pom.xml

pom.xml

单击此处查看POM文件

经过一些Google搜索后，我发现我们已经添加了hadoop auth jar.即使我发现了相同的错误

After some google I found I found we have add hadoop auth jar.even after i finding same error

推荐答案

我认为您正在打包旧的Hadoop jar.

I think you're packaging an old Hadoop jar.

看看storm-hdfs POM

Take a look at the storm-hdfs POM https://github.com/apache/storm/blob/v1.0.6/external/storm-hdfs/pom.xml. When you use the Shade plugin, the jar you end up with will contain all your dependencies, including transitive ones brought in through direct dependencies. Storm-hdfs declares a dependency on a list of Hadoop jars. You need to make sure that you're declaring the same list of Hadoop jars in your POM if you want to use a different version of Hadoop from the default.

具体情况是，您尚未在POM中声明hadoop-auth，因此POM与该jar的默认版本(2.6.1)打包在一起.由于该版本的hadoop-auth与其他Hadoop jar(2.9.1)不兼容，因此在运行时会出现异常.

Specifically what's happening is that you haven't declared hadoop-auth in your POM, so your POM gets packaged with the default version of that jar (2.6.1). Since that version of hadoop-auth is incompatible with the other Hadoop jars (which are 2.9.1), you get an exception at runtime.

您应该从导入的storm-hdfs中排除所有Hadoop jar，然后将要使用的jar放入Storm的lib目录中，或者将正确版本的Hadoop jar添加到POM中的依赖项列表中.

You should either exclude all Hadoop jars from your import of storm-hdfs and then put the jars you want to use in Storm's lib directory, or add the right versions of the Hadoop jars to your dependency list in your POM.

我想我找到了你的问题.您尚未将Storm-core的范围设置为提供.由于storm-core取决于hadoop-auth，并且您尚未明确声明，因此Maven将尝试根据依赖关系在树中的位置来猜测所需的hadoop-auth版本.由于hadoop-auth在某些Hadoop依赖项中显示为2.9.1，而在Storm-core中显示为2.6.1，因此您碰巧将2.6.1放入了jar.

I think I found your issue. You haven't set the scope of storm-core to provided. Since storm-core depends on hadoop-auth, and you haven't declared it explicitly, Maven will try to guess which version of hadoop-auth you need based on where the dependency appears in the tree. Since hadoop-auth appears as 2.9.1 through some of your Hadoop dependencies, but 2.6.1 through storm-core, you happen to get 2.6.1 put in your jar.

如果将来想避免这种情况，则应使用Maven的dependencyManagement块

If you want to avoid this kind of thing in the future, you should use Maven's dependencyManagement block https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management.

即您应该在pom中添加如下所示的内容，然后删除hadoop jars的排除项.

i.e. you should add something like the following to your pom, and then remove the exclusions of hadoop jars.

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-auth</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-common</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

这篇关于找不到:org.apache.hadoop.security.authentication.util.KerberosUtil的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！