Kafka 集群 - 生产者 [英] Kafka Cluster - Producer

问题描述

我有几个关于 Kafka 的问题.如果有人可以通过回复其中之一来帮助我，我将不胜感激.

I have several questions about Kafka. If someone can help me by responding to one of them, i will be very thankful.

先谢谢你:)

Q1) 我知道分区在 Kafka Broker 之间分开.但分裂是基于什么?.例如，如果我有 3 个代理和 6 个分区，如何确保每个代理将有 2 个分区?Kafka 目前是如何进行这种拆分的?

Q1) I know that partitions are split across Kafka Broker. But the split is based on what ?. For instance, if I have 3 brokers and 6 partitions, how to ensure that each broker will have 2 partitions ? How this split is currently made in Kafka ?

Q2) 当生产者发送一条新消息时，消息的默认格式是什么?Avro 格式 ?例如，如何将此默认格式更改为可能更合适的另一种格式?

Q2) When a producer send a new message, what id the default format of the message ? Avro format ? How can I change this default format to another format which may be more suitable for example ?

Q3) 我知道要配置分区内文件(日志段)的最大大小，我必须更改以下配置:log.segment.bytes(默认为 1G).但是哪个配置参数，我必须更改以增加/减少目录(即分区)的最大大小?

Q3) I know that to configure the maximum size of a file (log segment) within a partition, I have to change the following configuration : log.segment.bytes (1G by default). But which configuration parameter, I have to change to increase/decrease the maximum size of a directory (i.e a partition) ?

Q4) 如果一个分区被认为是领导者已经死了，那么跟随者分区中的一个将成为领导者.选举新领导人的步骤是什么?(i.e) 新领导者的选举是如何组成的?

Q4) If a partition consider as the leader is dead, one of the follower partition will take the lead. What is the step, to elect the new leader ? (i.e) How the election of a new leader is made of ?

Q5) 什么是配置参数，它允许我更改磁盘上 2 个持久化之间的时间?(在磁盘上持久化数据 - 顺序写入)

Q5) What is the configuration parameter, that allow me to change the time between 2 persist on disk ? (persist data on disk - sequential write)

Q6) 消息如何从 Kafka 代理的硬盘 Head 发送到 Kafka 消费者?Kafka Broker 和 Zookeeper 之间的交互是什么?是 Zookeeper 将消息发送给消费者还是 Kafka Broker?

Q6) How the message is sent from the hard disk Head of a Kafka broker to a Kafka consumer ? What is the interaction between Kafka Broker and Zookeeper ? Is it Zookeeper which send the message to the consumer or Kafka Broker ?

先谢谢你，

推荐答案

Q1:参见 如何将 Partitions 拆分为 Kafka Broker?

Q2:代理不知道消息格式——他们将消息视为纯字节数组.因此，它可以处理您想要的任何消息格式.格式由您自己的代码确定 - 选择您想要的任何内容，然后向生产者/消费者提供相应的反序列化程序.

Q2: Brokers are agnostic to the message format -- they treat messages a plain byte arrays. Thus, it can handle any message format you want to have. The format is determined in your own code -- choose whatever you want and just provide the corresponding de/serializer to the producer/consumer.

Q3:主题和分区会在经过可配置的保留时间 (log.retention.ms) 或增长超过 log.retention.bytes 后被截断.此外，可以压缩主题以避免无限增长.(参见log.cleanup.policy)

Q3: Topics and thus partitions are either truncated after a configurable retention time passed (log.retention.ms) or if they grow beyond log.retention.bytes. Furthermore, topics can be compacted to avoid infinite growth. (cf. log.cleanup.policy)

Q4:使用 Apache Zookeeper 进行领导人选举.

Q4: For leader election Apache Zookeeper is used.

Q5:不明白这个问题.

Q5: Don't understand the question.

Q6:ZK 仅用于维护元数据(例如哪些主题确实存在).ZK 不参与客户端-代理通信的任何实际数据传输.Kafka 使用自己的网络协议.有关更多详细信息，请参阅 Kafka Wiki:https://cwiki.apache.org/confluence/显示/KAFKA/索引

Q6: ZK is only used to maintain metadata (which topics do exists for example). ZK is not involved in any actual data transfer of client-broker communication. Kafka uses its own network protocol. See the Kafka Wiki for more details: https://cwiki.apache.org/confluence/display/KAFKA/Index

这篇关于Kafka 集群 - 生产者的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！