未找到:org.apache.hadoop.security.authentication.util.KerberosUtil [英] Not found :org.apache.hadoop.security.authentication.util.KerberosUtil

问题描述

我在集群中运行storm jar，在那里我配置了hadoop、kafka、storm集群

当我在本地模式下运行 jar 时它工作正常，当我在 Storm 集群上运行它时，我在 Storm UI 中发现了相应的错误:

java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosTicket(Ljavax/security/auth/Subject;)Z atorg.apache.hadoop.security.UserGroupInformation.(UserGroupInformation.java:666) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:861) atorg.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)

pom.xml

点击此处查看 POM 文件

经过一些谷歌我发现我发现我们已经添加了 hadoop auth jar.即使在我发现相同的错误之后

解决方案

我认为您正在打包一个旧的 Hadoop jar.

看一下storm-hdfs POM https://github.com/apache/storm/blob/v1.0.6/external/storm-hdfs/pom.xml.当您使用 Shade 插件时，您最终得到的 jar 将包含您的所有依赖项，包括通过直接依赖项引入的传递性依赖项.Storm-hdfs 声明了对 Hadoop jar 列表的依赖.如果要使用与默认版本不同的 Hadoop 版本，则需要确保在 POM 中声明了相同的 Hadoop jar 列表.

具体发生的情况是您尚未在 POM 中声明 hadoop-auth，因此您的 POM 与该 jar 的默认版本 (2.6.1) 打包在一起.由于该版本的 hadoop-auth 与其他 Hadoop jar(2.9.1)不兼容，因此您会在运行时遇到异常.

您应该从storm-hdfs 的导入中排除所有Hadoop jar，然后将您想要使用的jar 放在Storm 的lib 目录中，或者将正确版本的Hadoop jar 添加到您的POM 中的依赖项列表中.

我想我发现了你的问题.您尚未将storm-core 的范围设置为provided.由于storm-core 依赖于hadoop-auth，而您还没有明确声明它，Maven 将尝试根据依赖关系出现在树中的位置来猜测您需要哪个版本的hadoop-auth.由于 hadoop-auth 通过您的一些 Hadoop 依赖项显示为 2.9.1，但通过 Storm-core 显示为 2.6.1，因此您碰巧将 2.6.1 放入您的 jar 中.

如果以后想避免这种事情，应该使用Maven的dependencyManagement块https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management.

即您应该在 pom 中添加类似以下内容，然后删除 hadoop jars 的排除项.

<依赖项><依赖><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>${hadoop.version}</version></依赖><依赖><groupId>org.apache.hadoop</groupId><artifactId>hadoop-auth</artifactId><version>${hadoop.version}</version></依赖><依赖><groupId>org.apache.hadoop</groupId><artifactId>hadoop-common</artifactId><version>${hadoop.version}</version></依赖></依赖项></dependencyManagement>

I am running storm jar in a cluster ,where I configured hadoop,kafka,storm cluster

when I run the jar in local mode it works fine ,when I run it on storm cluster, I am finding respective error in Storm UI:

java.lang.NoSuchMethodError: org.apache.hadoop.security.authentication.util.KerberosUtil.hasKerberosTicket(Ljavax/security/auth/Subject;)Z at 
org.apache.hadoop.security.UserGroupInformation.<init>(UserGroupInformation.java:666) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:861) at 
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)

pom.xml

Click here to view POM file

After some google I found I found we have add hadoop auth jar.even after i finding same error

解决方案

I think you're packaging an old Hadoop jar.

Take a look at the storm-hdfs POM https://github.com/apache/storm/blob/v1.0.6/external/storm-hdfs/pom.xml. When you use the Shade plugin, the jar you end up with will contain all your dependencies, including transitive ones brought in through direct dependencies. Storm-hdfs declares a dependency on a list of Hadoop jars. You need to make sure that you're declaring the same list of Hadoop jars in your POM if you want to use a different version of Hadoop from the default.

Specifically what's happening is that you haven't declared hadoop-auth in your POM, so your POM gets packaged with the default version of that jar (2.6.1). Since that version of hadoop-auth is incompatible with the other Hadoop jars (which are 2.9.1), you get an exception at runtime.

You should either exclude all Hadoop jars from your import of storm-hdfs and then put the jars you want to use in Storm's lib directory, or add the right versions of the Hadoop jars to your dependency list in your POM.

Edit: I think I found your issue. You haven't set the scope of storm-core to provided. Since storm-core depends on hadoop-auth, and you haven't declared it explicitly, Maven will try to guess which version of hadoop-auth you need based on where the dependency appears in the tree. Since hadoop-auth appears as 2.9.1 through some of your Hadoop dependencies, but 2.6.1 through storm-core, you happen to get 2.6.1 put in your jar.

If you want to avoid this kind of thing in the future, you should use Maven's dependencyManagement block https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management.

i.e. you should add something like the following to your pom, and then remove the exclusions of hadoop jars.

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-auth</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-common</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

这篇关于未找到:org.apache.hadoop.security.authentication.util.KerberosUtil的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！