使用Kafka主题存储数据多年 [英] Use Kafka topics to store data for many years

问题描述

我正在寻找一种从多个设备收集指标数据的方法.数据应通过多个类似分组依据"的函数进行汇总.汇总功能列表不完整，以后将添加新的汇总，并且需要汇总从第一天开始收集的所有数据.

I am looking for a way of collecting metrics data from multiple devices. The data should be aggregated by multiple "group by" like functions. The aggregation functions list is not complete and new aggregations will be added later and it will be required to aggregate all data collected from first days.

创建具有100年有效期的Kafka主题并将其用作此目的的数据存储库是否可以?因此，新的汇总将能够从主题的开头读取，而现有的汇总将继续从其偏移量开始?

Is it fine to create Kafka topic with 100 year expiration period and use it as a datastore for this purpose? So new aggregations will be able to read from topic's start while existing aggregations will continue from their's offsets?

推荐答案

原则上，可以，因为您概述的原因，您可以使用Kafka进行长期存储-重新处理源数据以导出其他汇总/计算.

In principle, yes you can use Kafka for long-term storage, exactly for the reason you outline - reprocessing of source data to derive additional aggregates/calculations.

一些参考文献:

• https://www.confluent.io/blog/okay-store-data-apache-kafka/
• https://www.confluent.io/Blog/publishing-apache-kafka-new-york-times/

这篇关于使用Kafka主题存储数据多年的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！