如果zookeeper.set.acl设置为true，则Kafka无法启动 [英] Kafka not starting up if zookeeper.set.acl is set to true
问题描述
我有一个设置了kerberized的Zookeeper和一个设置了false的zookeeper的Kafka，它们可以很好地工作.当我尝试将参数设置为true来启动Kafka时，我在zookeeper日志中得到了此信息:
I have a set up of kerberized Zookeeper and kerberized Kafka which works fine with zookeeper.set.acl set to false. When I try to start Kafka with the parameter set to true, I get this in the zookeeper logs:
Nov 12 13:36:26 <zk host> docker:zookeeper_corelinux_<zk host>[1195]: [2019-11-12 13:36:26,625] INFO Client attempting to establish new session at /<kafka ip>:54272 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 12 13:36:26 <zk host> docker:zookeeper_corelinux_<zk host>[1195]: [2019-11-12 13:36:26,631] INFO Established session 0x3007c8bcb5c0000 with negotiated timeout 6000 for client /<kafka ip>:54272 (org.apache.zookeeper.server.ZooKeeperServer)
Nov 12 13:36:26 <zk host> docker:zookeeper_corelinux_<zk host>[1195]: [2019-11-12 13:36:26,775] INFO Successfully authenticated client: authenticationID=kafka/<kafka host>@REALM; authorizationID=kafka/<kafka host>@REALM. (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
Nov 12 13:36:26 <zk host> docker:zookeeper_corelinux_<zk host>[1195]: [2019-11-12 13:36:26,778] INFO Setting authorizedID: kafka (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
Nov 12 13:36:26 <zk host> docker:zookeeper_corelinux_<zk host>[1195]: [2019-11-12 13:36:26,778] INFO adding SASL authorization for authorizationID: kafka (org.apache.zookeeper.server.ZooKeeperServer)
Nov 12 13:36:26 <zk host> docker:zookeeper_corelinux_<zk host>[1195]: [2019-11-12 13:36:26,807] ERROR Missing AuthenticationProvider for sasl (org.apache.zookeeper.server.PrepRequestProcessor)
Nov 12 13:36:26 <zk host> docker:zookeeper_corelinux_<zk host>[1195]: [2019-11-12 13:36:26,808] INFO Got user-level KeeperException when processing sessionid:0x3007c8bcb5c0000 type:create cxid:0x4 zxid:0x100000005 txntype:-1 reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode = InvalidACL for /brokers/ids (org.apache.zookeeper.server.PrepRequestProcessor)
Nov 12 13:36:26 <zk host> docker:zookeeper_corelinux_<zk host>[1195]: [2019-11-12 13:36:26,829] INFO Processed session termination for sessionid: 0x3007c8bcb5c0000 (org.apache.zookeeper.server.PrepRequestProcessor)
Kafka和Zookeeper都在docker中运行(使用Confluent的图像)
Both Kafka and Zookeeper are running in docker (using Confluent's images)
这是Zookeeper配置(通过环境变量传递):
Here's the Zookeeper config (passed in via environment variables):
"ZOOKEEPER_AUTHPROVIDER_1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider",
"KAFKA_OPTS=-Djava.security.auth.login.config=/etc/zookeeper/secrets/zookeeper_jaas.conf -Dzookeeper.kerberos.removeHostFromPrincipal=true -Dzookeeper.kerberos.removeRealmFromPrincipal=true",
"ZOOKEEPER_SERVER_ID=1",
"ZOOKEEPER_REQUIRECLIENTAUTHSCHEME=SASL",
"KAFKA_JMX_HOSTNAME=<zk host>",
"ZOOKEEPER_INIT_LIMIT=10",
"ZOOKEEPER_JASSLOGINRENEW=3600000",
"ZOOKEEPER_LOG4J_PROP=DEBUG,ROLLINGFILE",
"ZOOKEEPER_MAX_CLIENT_CNXNS=0",
"ZOOKEEPER_SERVERS=0.0.0.0:2888:3888;zookeeper2:2888:3888;zookeeper3:2888:3888",
"ZOOKEEPER_DATA_DIR=/data/zookeeper",
"ZOOKEEPER_CLIENT_PORT=2181",
"KAFKA_JMX_PORT=55554"
Zookeeper JAAS:
Zookeeper JAAS:
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
doNotPrompt=true
useTicketCache=false
keyTab="/etc/zookeeper/secrets/kfkzkp.keytab"
principal="zookeeper/<zk host>@REALM";
};
这是Kafka配置:
"KAFKA_ZOOKEEPER_SET_ACL=true",
"KAFKA_DEFAULT_REPLICATION_FACTOR=3",
"KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL=GSSAPI",
"KAFKA_ADVERTISED_LISTENERS=SASL_SSL://<kafka host>:9092",
"KAFKA_OPTS=-Djava.security.auth.login.config=/etc/kafka/secrets/kafka_server_jaas.conf",
"KAFKA_ZOOKEEPER_CONNECT=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181",
"KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND=true",
"KAFKA_SSL_CLIENT_AUTH=required",
"KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE=False",
"KAFKA_LOG_DIRS=/data/kafka",
"KAFKA_SASL_KERBEROS_SERVICE_NAME=kafka",
"KAFKA_SSL_TRUSTSTORE_FILENAME=root-ca-certificate.jks",
"KAFKA_JMX_HOSTNAME=<kafka host>",
"KAFKA_MIN_INSYNC_REPLICAS=2",
"KAFKA_JMX_PORT=55555",
"KAFKA_SSL_KEY_CREDENTIALS=redacted",
"KAFKA_AUTHORIZER_CLASS_NAME=kafka.security.auth.SimpleAclAuthorizer",
"KAFKA_SUPER_USERS=User:superuser;User:me",
"KAFKA_SSL_KEYSTORE_FILENAME=<kafka host>.jks",
"KAFKA_SSL_KEYSTORE_CREDENTIALS=redacted",
"KAFKA_SSL_TRUSTSTORE_CREDENTIALS=redacted",
"KAFKA_AUTO_CREATE_TOPICS_ENABLE=true",
"KAFKA_SASL_ENABLED_MECHANISMS=GSSAPI,PLAIN",
"KAFKA_LISTENERS=SASL_SSL://<kafka host>:9092",
"KAFKA_SECURITY_INTER_BROKER_PROTOCOL=SASL_SSL",
Kafka JAAS:
Kafka JAAS:
// Zookeeper client authentication
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
doNotPrompt=true
useTicketCache=false
serviceName=kafka
keyTab="/etc/kafka/secrets/kfkzkp.keytab"
principal="kafka/<kafka host>@REALM";
};
我已经看了一段时间了，并浏览了Google上的大多数相关内容(包括stackoverflow的一些链接).任何建议都将受到欢迎.
I have been looking at this for a while now and have gone through most of the relevant stuff on google (including a few links from stackoverflow). Any suggestions would be most welcome.
推荐答案
弄清楚了.由于某些原因，某些变量无法从环境中正确提取.我昨天用ZOOKEEPER_KERBEROS_REMOVEREALMFROMPRINCIPAL(和REMOVEHOSTFROMPRINCIPAL)注意到了这一点.所以我尝试移动这些
Figured it out. For some reason, some variables aren't picked up correctly from the environment. I noticed this yesterday with ZOOKEEPER_KERBEROS_REMOVEREALMFROMPRINCIPAL (and REMOVEHOSTFROMPRINCIPAL). So I tried moving these
"ZOOKEEPER_AUTHPROVIDER_1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider",
"ZOOKEEPER_REQUIRECLIENTAUTHSCHEME=SASL",
进入
KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/zookeeper/secrets/zookeeper_jaas.conf -Dzookeeper.kerberos.removeHostFromPrincipal=true -Dzookeeper.kerberos.removeRealmFromPrincipal=true -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dzookeeper.requireClientAuthScheme=sasl"
然后排序.
这篇关于如果zookeeper.set.acl设置为true，则Kafka无法启动的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
