Cloudera上的Kafka-test = TOPIC_AUTHORIZATION_FAILED [英] Kafka on Cloudera - test=TOPIC_AUTHORIZATION_FAILED

问题描述

我们刚刚从CDH 5.3.6升级到5.10.0，并在尝试写入Kafka主题时开始出现错误。我们在所有内容上均具有默认设置，未启用SSL或Kerberos身份验证。当使用控制台生产者写我的主题之一时，出现以下错误：

We just upgraded from CDH 5.3.6 to 5.10.0, and started getting errors when trying to write to Kafka topics. We have the default settings on everything, no SSL or Kerberos authentication enabled. When use the console producer to write to one of my topics, I get this error:

/ usr / bin / kafka-console-producer --broker-list = myhost1 .dev.com：9092，myhost2.dev.com：9092 --topic测试

/usr/bin/kafka-console-producer --broker-list=myhost1.dev.com:9092,myhost2.dev.com:9092 --topic test

17/03/06 21:00:57 INFO utils.AppInfoParser: Kafka version : 0.10.0-kafka-2.1.0
17/03/06 21:00:57 INFO utils.AppInfoParser: Kafka commitId : unknown
x
17/03/06 21:00:59 WARN clients.NetworkClient: Error while fetching metadata with correlation id 0 : {test=TOPIC_AUTHORIZATION_FAILED}

看着/ var / log / kafka /，我看到了很多这样的例外：

Looking at /var/log/kafka/, I see a bunch of these exceptions:

2017-03-06 21:00:26,964 WARN org.apache.sentry.provider.common.HadoopGroupMappingService: Unable to obtain groups for ANONYMOUS
java.io.IOException: No groups found for user ANONYMOUS
    at org.apache.hadoop.security.Groups.noGroupsForUser(Groups.java:190)
    at org.apache.hadoop.security.Groups.getGroups(Groups.java:210)
    at org.apache.sentry.provider.common.HadoopGroupMappingService.getGroups(HadoopGroupMappingService.java:60)
    at org.apache.sentry.provider.common.ResourceAuthorizationProvider.getGroups(ResourceAuthorizationProvider.java:167)
    at org.apache.sentry.provider.common.ResourceAuthorizationProvider.doHasAccess(ResourceAuthorizationProvider.java:97)
    at org.apache.sentry.provider.common.ResourceAuthorizationProvider.hasAccess(ResourceAuthorizationProvider.java:91)
    at org.apache.sentry.kafka.binding.KafkaAuthBinding.authorize(KafkaAuthBinding.java:212)
    at org.apache.sentry.kafka.authorizer.SentryKafkaAuthorizer.authorize(SentryKafkaAuthorizer.java:63)
    at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$authorize$2.apply(KafkaApis.scala:321)
    at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$authorize$2.apply(KafkaApis.scala:321)
    at scala.Option.map(Option.scala:146)
    at kafka.server.KafkaApis.kafka$server$KafkaApis$$authorize(KafkaApis.scala:321)
    at kafka.server.KafkaApis$$anonfun$30.apply(KafkaApis.scala:702)
    at kafka.server.KafkaApis$$anonfun$30.apply(KafkaApis.scala:702)
    at scala.collection.TraversableLike$$anonfun$partition$1.apply(TraversableLike.scala:314)
    at scala.collection.TraversableLike$$anonfun$partition$1.apply(TraversableLike.scala:314)
    at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
    at scala.collection.TraversableLike$class.partition(TraversableLike.scala:314)
    at scala.collection.AbstractTraversable.partition(Traversable.scala:104)
    at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:702)
    at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
    at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
    at java.lang.Thread.run(Thread.java:745)

我一直在寻找解决方案，但空无一人 至今。我需要将匿名用户分配给某处的某些组吗？我可以在CDH 5.3.6中为我的主题写消息，但是升级似乎出现了问题。

I've been looking for a solution to this, but have come up empty so far. Do I need to assign the ANONYMOUS user to some groups somewhere? I was able to write messages to my topics in CDH 5.3.6, but it appears something has gone wrong in the upgrade.

只是尝试获取helloWorld / Quickstart示例升级到CDH 5.10.0后，可以在DEV Kafka上再次工作。

Just trying to get the helloWorld/Quickstart example to work again on our DEV Kafka after upgrading to CDH 5.10.0.

-----------------临时解决方法---

----------------- Temporary workaround solution ---

在cloudera manager 5.10中，kafka配置中有一个super.users属性。将匿名添加到该列表中，使我可以从自己的主题中产生和消费。

In cloudera manager 5.10 there is a super.users property in the kafka configuration. Adding ANONYMOUS to that list, allowed me to produce and consume from my topics.

我已经在/opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/etc/kafka/conf.dist中尝试过/server.properties，这无效。因此，Cloudera必须在其他地方管理这些值。

I had already tried doing this in /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/etc/kafka/conf.dist/server.properties, which had no effect. So Cloudera must be managing these values elsewhere.

推荐答案

Kafka严格区分身份验证和授权-即使您通过Curb或启用S​​SL仍可以通过以下参数打开授权：

Kafka strictly distinguishes between authentication and authorization - even if you have authentication via Kerb or SSL turned of it is still possible to turn authorization on via the following parameter:

authorizer.class.name=kafka.security.auth.SimpleAclAuthorize‌​r

这将使Kafka每次访问都检查ACL，因为已启用身份验证但是，如果您的情况是这样，则如果未为此用户设置ACL，则每个用户都将被评估为 ANONYMOUS 并被拒绝。

This will make Kafka check ACLs for every access - since authentication is turned of in your case though, every user will be evaluated as ANONYMOUS and denied, if there are no ACLs set for this user.

您可以删除该用户从您的配置中进行设置，这应该会使Kafka恢复到其古老的，信任的自我。我不确定您在Cloudera Manager中的执行位置，因此可以选择将 ANONYMOUS 添加到CM中可用的超级用户列表中。或者当然只定义一个ACL以允许访问 ANONYMOUS 。

You can delete that setting from your config which should make Kafka return to its old, trusting, self. I am not sure where you'd do this in Cloudera Manager though, so an alternative would be to add ANONYMOUS to the list of super users which is available in CM. Or of course just define an ACL to allow access to ANONYMOUS.

对于以后的生产使用，您可能应该设置SSL或Kerberos并如果有可能从外部访问群集，则定义适当的ACL。

For production use later on you should probably set up either SSL or Kerberos and define proper ACLs if there is any chance of the cluster being accessed from the outside.

这篇关于Cloudera上的Kafka-test = TOPIC_AUTHORIZATION_FAILED的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！