使用Kafka主题存储多年数据 [英] Use Kafka topics to store data for many years

问题描述

我正在寻找一种从多个设备收集指标数据的方法.数据应该由多个分组依据"之类的函数聚合.聚合函数列表不完整，稍后会添加新的聚合，并且需要聚合第一天收集的所有数据.

I am looking for a way of collecting metrics data from multiple devices. The data should be aggregated by multiple "group by" like functions. The aggregation functions list is not complete and new aggregations will be added later and it will be required to aggregate all data collected from first days.

为此目的创建具有 100 年有效期的 Kafka 主题并将其用作数据存储是否可以?那么新的聚合将能够从主题的开始读取，而现有的聚合将继续从它们的偏移量开始?

Is it fine to create Kafka topic with 100 year expiration period and use it as a datastore for this purpose? So new aggregations will be able to read from topic's start while existing aggregations will continue from their's offsets?

推荐答案

原则上，是的，您可以使用 Kafka 进行长期存储，这正是您概述的原因 - 重新处理源数据以获得额外的聚合/计算.

In principle, yes you can use Kafka for long-term storage, exactly for the reason you outline - reprocessing of source data to derive additional aggregates/calculations.

几个参考:

• https://www.confluent.io/blog/OK-store-data-apache-kafka/
• https://www.confluent.io/博客/publishing-apache-kafka-new-york-times/

这篇关于使用Kafka主题存储多年数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！