Kafka Connect BigQuery Sink 连接器从架构注册表请求不正确的主题名称 [英] Kafka Connect BigQuery Sink Connector requests incorrect subject names from the Schema Registry

查看:29
本文介绍了Kafka Connect BigQuery Sink 连接器从架构注册表请求不正确的主题名称的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

在我们的 Kafka 上尝试使用 confluentinc/kafka-connect-bigquery 时(Avro) 事件,我遇到以下错误:

While trying to use confluentinc/kafka-connect-bigquery on our Kafka (Avro) events, I run into the following error:

org.apache.kafka.connect.errors.DataException: Failed to deserialize data for topic domain.user to Avro: 
    at io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:125)
[...]
Caused by: org.apache.kafka.common.errors.SerializationException: Error retrieving Avro key schema version for id 619
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Subject 'domain.user-key' not found.; error code: 40401
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:295)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:355)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.lookUpSubjectVersion(RestService.java:440)
[...]

是的,模式注册表中没有主题 domain.user-key:

And yes, there is no subject domain.user-key in the schema registry:

curl --silent -X GET http://avro-schema-registry.core-kafka.svc.cluster.local:8081/subjects | jq . | grep "domain\.user"

[...]
  "domain.user-com.acme.message_schema.type.domain.key.DefaultKey",
  "domain.user-com.acme.message_schema.domain.user.Key",
[...]

如何让连接器使用正确的主题名称?

How can I make the connector use the right subject names?

我的 properties/connector.properties(我正在使用 quickstart 文件夹.)如下所示:

My properties/connector.properties (I'm using the quickstart folder.) looks as follows:

[...]
topics=domain.user
sanitizeTopics=true
autoUpdateSchemas=true
autoCreateTables=true
allowNewBigQueryFields=true
[...]

最后,我想使用 topics.regex=domain.* 而不是 topics=domain.user 来捕获我们所有的域事件主题,但是使用这我得到了相同类型的错误(只是针对不同的主题).

In the end, I want to use topics.regex=domain.* instead of topics=domain.user to capture all our domain-event topics, but with this I get the same type of error (just for a different subject).

推荐答案

您需要设置key.converter.key.subject.name.strategy &value.converter.value.subject.name.strategyio.confluent.kafka.serializers.subject.TopicRecordNameStrategy

You need to set key.converter.key.subject.name.strategy & value.converter.value.subject.name.strategy to be io.confluent.kafka.serializers.subject.TopicRecordNameStrategy

这篇关于Kafka Connect BigQuery Sink 连接器从架构注册表请求不正确的主题名称的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
相关文章
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆