将Spark SQL数据集用作基于RDD的作业 [英] Consume Spark SQL dataset as RDD based job

问题描述

Spark数据帧具有toRDD()方法，但我不明白它的用处.我们是否可以通过将转换后的源数据集处理为RDD而不是创建并启动DataStreamWriter来启动SQL流工作?

Spark dataframe have toRDD() method but I don't understand how It's useful. Can we start a SQL streaming job by processing converted source dataset to RDD instead of making and starting DataStreamWriter?

推荐答案

Dataset为批处理和流处理提供统一的API，并非每种方法都适用于流数据集.如果仔细搜索，将会发现流式数据集无法使用的其他方法(例如describe).

Dataset provides uniform API for both batch and streaming processing and not every method is applicable to streaming Datasets. If you search carefully, you'll find other methods which cannot be used with streaming Datasets (for example describe).

我们可以通过将转换后的源数据集处理为RDD而不是创建并启动DataStreamWriter来启动SQL流工作吗?

Can we start a SQL streaming job by processing converted source dataset to RDD instead of making and starting DataStreamWriter?

我们不能.从结构化流媒体开始，而在结构化流媒体中保持不变.不允许转换为RDD.

We cannot. What starts in structured streaming, stays in structured streaming. Conversions to RDD are not allowed.

这篇关于将Spark SQL数据集用作基于RDD的作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！