Kafka Connect BigQuery Sink Connector从架构注册表请求不正确的主题名称 [英] Kafka Connect BigQuery Sink Connector requests incorrect subject names from the Schema Registry
问题描述
尝试在我们的Kafka上使用 confluentinc/kafka-connect-bigquery (Avro)事件,我遇到以下错误:
While trying to use confluentinc/kafka-connect-bigquery on our Kafka (Avro) events, I run into the following error:
org.apache.kafka.connect.errors.DataException: Failed to deserialize data for topic domain.user to Avro:
at io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:125)
[...]
Caused by: org.apache.kafka.common.errors.SerializationException: Error retrieving Avro key schema version for id 619
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Subject 'domain.user-key' not found.; error code: 40401
at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:295)
at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:355)
at io.confluent.kafka.schemaregistry.client.rest.RestService.lookUpSubjectVersion(RestService.java:440)
[...]
是的,架构注册表中没有主题 domain.user-key
:
And yes, there is no subject domain.user-key
in the schema registry:
curl --silent -X GET http://avro-schema-registry.core-kafka.svc.cluster.local:8081/subjects | jq . | grep "domain\.user"
[...]
"domain.user-com.acme.message_schema.type.domain.key.DefaultKey",
"domain.user-com.acme.message_schema.domain.user.Key",
[...]
如何使连接器使用正确的主题名称?
How can I make the connector use the right subject names?
我的 properties/connector.properties
(我正在使用
My properties/connector.properties
(I'm using the quickstart folder.) looks as follows:
[...]
topics=domain.user
sanitizeTopics=true
autoUpdateSchemas=true
autoCreateTables=true
allowNewBigQueryFields=true
[...]
最后,我想使用 topics.regex = domain.*
而不是 topics = domain.user
来捕获我们所有的域事件主题,但是我会得到相同类型的错误(只是针对不同的主题).
In the end, I want to use topics.regex=domain.*
instead of topics=domain.user
to capture all our domain-event topics, but with this I get the same type of error (just for a different subject).
推荐答案
您需要设置 key.converter.key.subject.name.strategy
& value.converter.value.subject.name.strategy
成为 io.confluent.kafka.serializers.subject.TopicRecordNameStrategy
You need to set key.converter.key.subject.name.strategy
& value.converter.value.subject.name.strategy
to be io.confluent.kafka.serializers.subject.TopicRecordNameStrategy
这篇关于Kafka Connect BigQuery Sink Connector从架构注册表请求不正确的主题名称的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!