Kafka:消费者API与流API [英] Kafka: Consumer API vs Streams API

问题描述

我最近开始学习Kafka，并遇到了这些问题.

I recently started learning Kafka and end up with these questions.

1. Consumer和Stream之间有什么区别?对我而言，如果有任何工具/应用程序消费来自Kafka的消息，则是Kafka世界中的消费者.

1. What is the difference between Consumer and Stream? For me, if any tool/application consume messages from Kafka is a consumer in the Kafka world.

Stream有何不同之处，因为它也会从Kafka中消费或产生消息到Kafka?以及为什么需要它，因为我们可以编写自己的消费者 使用消费者API的应用程序并根据需要对其进行处理，还是将其从消费者应用程序发送到Spark?

How Stream is different as this also consumes from or produce messages to Kafka? and why is it needed as we can write our own consumer application using Consumer API and process them as needed or send them to Spark from the consumer application?

我在此方面使用过Google，但没有得到任何好的答案.抱歉，这个问题太琐碎了.

I did Google on this, but did not get any good answers for this. Sorry if this question is too trivial.

推荐答案

2018年4月9日更新:如今，您还可以使用

Update April 09, 2018: Nowadays you can also use ksqlDB, the event streaming database for Kafka, to process your data in Kafka. ksqlDB is built on top of Kafka's Streams API, and it too comes with first-class support for "streams" and "tables".

Consumer API和Streams API有什么区别?

what is the difference between Consumer API and Streams API?

Kafka的Streams API( https://kafka.apache.org/documentation/streams/)是建立在Kafka的生产者和消费者客户之上的. 它比Kafka消费者客户端功能强大得多，表达能力也更强.以下是Kafka Streams API的一些功能:

Kafka's Streams API (https://kafka.apache.org/documentation/streams/) is built on top of Kafka's producer and consumer clients. It's significantly more powerful and also more expressive than the Kafka consumer client. Here are some of the features of the Kafka Streams API:

• 支持一次处理语义(Kafka版本0.11 +)
• 支持容错的有状态(当然也包括无状态)处理，包括流事件时间处理以及基于处理时间和均具有一流的支持流和表，这是流处理与数据库相遇的地方；实际上，大多数流处理应用程序都需要流和表来实现它们各自的用例，因此，如果流处理技术缺少这两种抽象之一(例如，不支持表)，则您将陷入困境或必须自己手动实现此功能. (祝你好运...)
• 支持交互式查询(也称为可查询状态")，以将最新的处理结果公开给其他应用程序和服务
• 更具表现力:它附带(1)一种功能编程风格

• Supports exactly-once processing semantics (Kafka versions 0.11+)
• Supports fault-tolerant stateful (as well as stateless, of course) processing including streaming joins, aggregations, and windowing. In other words, it supports management of your application's processing state out-of-the-box.
• Supports event-time processing as well as processing based on processing-time and ingestion-time
• Has first-class support for both streams and tables, which is where stream processing meets databases; in practice, most stream processing applications need both streams AND tables for implementing their respective use cases, so if a stream processing technology lacks either of the two abstractions (say, no support for tables) you are either stuck or must manually implement this functionality yourself (good luck with that...)
• Supports interactive queries (also called 'queryable state') to expose the latest processing results to other applications and services
• Is more expressive: it ships with (1) a functional programming style DSL with operations such as map, filter, reduce as well as (2) an imperative style Processor API for e.g. doing complex event processing (CEP), and (3) you can even combine the DSL and the Processor API.

请参见 http://docs.confluent.io/current/streams/introduction. html ，以获取有关Kafka Streams API的更详细但仍是高级的介绍，它还应该帮助您了解与较低级别的Kafka消费者客户端的区别.还有一个基于Docker的教程Kafka Streams API ，它本周早些时候我写了博客.

See http://docs.confluent.io/current/streams/introduction.html for a more detailed but still high-level introduction to the Kafka Streams API, which should also help you to understand the differences to the lower-level Kafka consumer client. There's also a Docker-based tutorial for the Kafka Streams API, which I blogged about earlier this week.

那么Kafka Streams API有何不同，因为它也会从Kafka中消费或产生消息?

So how is the Kafka Streams API different as this also consumes from or produce messages to Kafka?

是的，Kafka Streams API既可以读取数据，也可以将数据写入Kafka.

Yes, the Kafka Streams API can both read data as well as write data to Kafka.

为什么需要它，因为我们可以使用消费者API编写自己的消费者应用程序并根据需要进行处理，或将其从消费者应用程序发送到Spark?

and why is it needed as we can write our own consumer application using Consumer API and process them as needed or send them to Spark from the consumer application?

是的，您可以编写自己的消费者应用程序-正如我提到的那样，Kafka Streams API本身使用的是Kafka消费者客户端(加上生产者客户端)，但是您必须手动实现Streams API提供.有关免费"获得的所有内容，请参见上面的列表.因此，用户选择低级别的消费者客户端而不是功能更强大的Kafka Streams API的情况很少见.

Yes, you could write your own consumer application -- as I mentioned, the Kafka Streams API uses the Kafka consumer client (plus the producer client) itself -- but you'd have to manually implement all the unique features that the Streams API provides. See the list above for everything you get "for free". It is thus rather a rare circumstance that a user would pick the low-level consumer client rather than the more powerful Kafka Streams API.

这篇关于Kafka:消费者API与流API的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！