kafka-streams加入产生重复项 [英] kafka-streams join produce duplicates

问题描述

我有两个主题:

// photos
{'id': 1, 'user_id': 1, 'url': 'url#1'},
{'id': 2, 'user_id': 2, 'url': 'url#2'},
{'id': 3, 'user_id': 2, 'url': 'url#3'}

// users
{'id': 1, 'name': 'user#1'},
{'id': 1, 'name': 'user#1'},
{'id': 1, 'name': 'user#1'}

我由用户创建地图照片

KStream<Integer, Photo> photo_by_user = ...

photo_by_user.to("photo_by_user")

然后，我尝试连接两个表:

Then, I try to join two tables:

KTable<Integer, User> users_table = builder.table("users");
KTable<Integer, Photo> photo_by_user_table = builder.table("photo_by_user");
KStream<Integer, Result> results = users_table.join(photo_by_user_table, (a, b) -> Result.from(a, b)).toStream();

results.to("results");

结果类似

{'photo_id': 1, 'user': 1, 'url': 'url#1', 'name': 'user#1'}
{'photo_id': 2, 'user': 2, 'url': 'url#2', 'name': 'user#2'}
{'photo_id': 3, 'user': 3, 'url': 'url#3', 'name': 'user#3'}
{'photo_id': 1, 'user': 1, 'url': 'url#1', 'name': 'user#1'}
{'photo_id': 2, 'user': 2, 'url': 'url#2', 'name': 'user#2'}
{'photo_id': 3, 'user': 3, 'url': 'url#3', 'name': 'user#3'}

我看到结果重复了.为什么以及如何避免呢?

I see that results are duplicated. Why, and how to avoid it?

推荐答案

您可能会遇到一个已知的错误.在刷新"状态下，KTable-KTable联接可能会产生一些重复.请注意，严格意义上讲，这些重复项并不正确，因为结果是更新流，并且将"A"更新为"A"不会更改结果.当然，不希望得到这些重复项.尝试禁用缓存-如果不进行缓存，则不应发生刷新问题".

You might hit a known bug. On "flush" KTable-KTable join might produce some duplicates. Note, that those duplicates are strictly speaking not incorrect, because the result is an update-stream and updating "A" to "A" does not change the result. It's of course undesired to get those duplicates. Try to disable caching -- without caching, the "flush issues" should not occur.

这篇关于kafka-streams加入产生重复项的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！