Cloudera 上的 Kafka - test=TOPIC_AUTHORIZATION_FAILED [英] Kafka on Cloudera - test=TOPIC_AUTHORIZATION_FAILED

问题描述

我们刚刚从 CDH 5.3.6 升级到 5.10.0，并且在尝试写入 Kafka 主题时开始出现错误.我们对一切都有默认设置，没有启用 SSL 或 Kerberos 身份验证.当使用控制台生产者写入我的主题之一时，我收到此错误:

We just upgraded from CDH 5.3.6 to 5.10.0, and started getting errors when trying to write to Kafka topics. We have the default settings on everything, no SSL or Kerberos authentication enabled. When use the console producer to write to one of my topics, I get this error:

/usr/bin/kafka-console-producer --broker-list=myhost1.dev.com:9092,myhost2.dev.com:9092 --topic 测试

/usr/bin/kafka-console-producer --broker-list=myhost1.dev.com:9092,myhost2.dev.com:9092 --topic test

17/03/06 21:00:57 INFO utils.AppInfoParser: Kafka version : 0.10.0-kafka-2.1.0
17/03/06 21:00:57 INFO utils.AppInfoParser: Kafka commitId : unknown
x
17/03/06 21:00:59 WARN clients.NetworkClient: Error while fetching metadata with correlation id 0 : {test=TOPIC_AUTHORIZATION_FAILED}

查看/var/log/kafka/，我看到了一堆这样的异常:

Looking at /var/log/kafka/, I see a bunch of these exceptions:

2017-03-06 21:00:26,964 WARN org.apache.sentry.provider.common.HadoopGroupMappingService: Unable to obtain groups for ANONYMOUS
java.io.IOException: No groups found for user ANONYMOUS
    at org.apache.hadoop.security.Groups.noGroupsForUser(Groups.java:190)
    at org.apache.hadoop.security.Groups.getGroups(Groups.java:210)
    at org.apache.sentry.provider.common.HadoopGroupMappingService.getGroups(HadoopGroupMappingService.java:60)
    at org.apache.sentry.provider.common.ResourceAuthorizationProvider.getGroups(ResourceAuthorizationProvider.java:167)
    at org.apache.sentry.provider.common.ResourceAuthorizationProvider.doHasAccess(ResourceAuthorizationProvider.java:97)
    at org.apache.sentry.provider.common.ResourceAuthorizationProvider.hasAccess(ResourceAuthorizationProvider.java:91)
    at org.apache.sentry.kafka.binding.KafkaAuthBinding.authorize(KafkaAuthBinding.java:212)
    at org.apache.sentry.kafka.authorizer.SentryKafkaAuthorizer.authorize(SentryKafkaAuthorizer.java:63)
    at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$authorize$2.apply(KafkaApis.scala:321)
    at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$authorize$2.apply(KafkaApis.scala:321)
    at scala.Option.map(Option.scala:146)
    at kafka.server.KafkaApis.kafka$server$KafkaApis$$authorize(KafkaApis.scala:321)
    at kafka.server.KafkaApis$$anonfun$30.apply(KafkaApis.scala:702)
    at kafka.server.KafkaApis$$anonfun$30.apply(KafkaApis.scala:702)
    at scala.collection.TraversableLike$$anonfun$partition$1.apply(TraversableLike.scala:314)
    at scala.collection.TraversableLike$$anonfun$partition$1.apply(TraversableLike.scala:314)
    at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
    at scala.collection.TraversableLike$class.partition(TraversableLike.scala:314)
    at scala.collection.AbstractTraversable.partition(Traversable.scala:104)
    at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:702)
    at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
    at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
    at java.lang.Thread.run(Thread.java:745)

我一直在寻找解决方案，但到目前为止还没有找到.我是否需要将 ANONYMOUS 用户分配给某处的某些组?我可以在 CDH 5.3.6 中向我的主题写入消息，但升级过程中似乎出现了问题.

I've been looking for a solution to this, but have come up empty so far. Do I need to assign the ANONYMOUS user to some groups somewhere? I was able to write messages to my topics in CDH 5.3.6, but it appears something has gone wrong in the upgrade.

在升级到 CDH 5.10.0 后，只是试图让 helloWorld/Quickstart 示例在我们的 DEV Kafka 上再次运行.

Just trying to get the helloWorld/Quickstart example to work again on our DEV Kafka after upgrading to CDH 5.10.0.

----------------- 临时解决方案 ---

----------------- Temporary workaround solution ---

在 cloudera manager 5.10 中，kafka 配置中有一个 super.users 属性.将 ANONYMOUS 添加到该列表中，让我能够从我的主题中生产和消费.

In cloudera manager 5.10 there is a super.users property in the kafka configuration. Adding ANONYMOUS to that list, allowed me to produce and consume from my topics.

我已经尝试在/opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/etc/kafka/conf.dist/server.properties 中执行此操作，但没有任何效果.因此 Cloudera 必须在别处管理这些值.

I had already tried doing this in /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/etc/kafka/conf.dist/server.properties, which had no effect. So Cloudera must be managing these values elsewhere.

推荐答案

Kafka 严格区分认证和授权——即使你通过 Curb 或 SSL 开启认证，仍然可以通过以下参数开启授权:

Kafka strictly distinguishes between authentication and authorization - even if you have authentication via Kerb or SSL turned of it is still possible to turn authorization on via the following parameter:

authorizer.class.name=kafka.security.auth.SimpleAclAuthorize‌​r

这将使 Kafka 检查每次访问的 ACL - 因为在您的情况下已关闭身份验证，如果没有为该用户设置 ACL，每个用户都将被评估为 ANONYMOUS 并被拒绝.

This will make Kafka check ACLs for every access - since authentication is turned of in your case though, every user will be evaluated as ANONYMOUS and denied, if there are no ACLs set for this user.

您可以从配置中删除该设置，这将使 Kafka 恢复到旧的、信任的自我.不过，我不确定您会在 Cloudera Manager 中的何处执行此操作，因此另一种方法是将 ANONYMOUS 添加到 CM 中可用的超级用户列表中.或者当然只是定义一个 ACL 以允许访问 ANONYMOUS.

You can delete that setting from your config which should make Kafka return to its old, trusting, self. I am not sure where you'd do this in Cloudera Manager though, so an alternative would be to add ANONYMOUS to the list of super users which is available in CM. Or of course just define an ACL to allow access to ANONYMOUS.

对于稍后的生产使用，您可能应该设置 SSL 或 Kerberos 并定义适当的 ACL，如果有可能从外部访问集群.

For production use later on you should probably set up either SSL or Kerberos and define proper ACLs if there is any chance of the cluster being accessed from the outside.

这篇关于Cloudera 上的 Kafka - test=TOPIC_AUTHORIZATION_FAILED的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！