在不停止进程的情况下刷新 Spark 实时流中的数据帧 [英] Refresh Dataframe in Spark real-time Streaming without stopping process

问题描述

在我的应用程序中，我从 Kafka 队列获得了一个帐户流(使用 Spark 流和 kafka)

in my application i get a stream of accounts from Kafka queue (using Spark streaming with kafka)

而且我需要从 S3 获取与这些帐户相关的属性，因此我计划缓存 S3 结果数据帧，因为 S3 数据目前至少不会更新一天，将来可能会更改为 1 小时或 10 分钟.所以问题是如何在不停止进程的情况下定期刷新缓存的数据帧.

And i need to fetch attributes related to these accounts from S3 so im planning to cache S3 resultant dataframe as the S3 data will not updated atleast for a day for now, it might change to 1hr or 10 mins very soon in future .So the question is how can i refresh the cached dataframe periodically without stopping process.

**更新:我计划在 S3 中有更新时将事件发布到 kafka，使用 SNS 和 AWS lambda，我的流应用程序将订阅该事件并基于此事件刷新缓存的数据帧(基本上是不持久的() 缓存并从 S3 重新加载)这是一个好方法吗?

**Update:Im planning to publish an event into kafka whenever there is an update in S3, using SNS and AWS lambda and my streaming application will subscribe to the event and refresh the cached dataframe based on this event (basically unpersist()cache and reload from S3) Is this a good approach ?

推荐答案

这个问题最近在 Spark 邮件列表

据我所知，唯一的方法就是在新数据到达时从 S3 重新加载 DataFrame，这意味着您还必须重新创建流式 DF 并重新启动查询.这是因为 DataFrame 从根本上是不可变的.

As far as I know the only way to do what you're asking is to reload the DataFrame from S3 when new data arrives which means you have to recreate the streaming DF as well and restart the query. This is because DataFrames are fundamentally immutable.

如果您想在不重新加载的情况下更新(变异)DataFrame 中的数据，则需要尝试其中一种 与 Spark 集成或连接到 Spark 并允许更改的数据存储.我所知道的一个是 SnappyData.

If you want to update (mutate) data in a DataFrame without reloading it, you need to try one of the datastores that integrate with or connect to Spark and allow mutations. One that I'm aware of is SnappyData.

这篇关于在不停止进程的情况下刷新 Spark 实时流中的数据帧的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！