使用Python Spark Direct方法时如何从检查点恢复? [英] How to recover from checkpoint when using python spark direct approach?

问题描述

在阅读官方文档之后，我尝试使用 getOrCreate > checkpoint .一些摘要:

After read official docs, i tried using checkpoint with getOrCreate in spark streaming. Some snippets:

def get_ssc():
    sc = SparkContext("yarn-client")
    ssc = StreamingContext(sc, 10)  # calc every 10s
    ks = KafkaUtils.createDirectStream(
        ssc, ['lucky-track'], {"metadata.broker.list": KAFKA_BROKER})
    process_data(ks)

    ssc.checkpoint(CHECKPOINT_DIR)
    return ssc

if __name__ == '__main__':
    ssc = StreamingContext.getOrCreate(CHECKPOINT_DIR, get_ssc)

    ssc.start()
    ssc.awaitTermination()

该代码可以很好地进行恢复，但是已恢复的上下文始终可以在旧的流程函数上运行.这意味着，即使我更改了map/reduce函数代码，也根本无法正常工作.

The code works fine for recover, but the recovered context always works on the old process function. It means that even if i changed map/reduce function code, it not works at all.

直到现在，spark(1.5.2)仍不支持python的任意偏移量.那么，我应该怎么做才能使其正常工作?

Until now, spark(1.5.2) still not support arbitrary offset for python. So, what should i do to make this work properly?

推荐答案

这种行为是设计使然"的，并且对于java/scala Spark应用程序也有效.在检查点时将整个代码序列化.如果代码更改，则检查点数据应被截断.

Such behaviour is "by design", and is valid also for java/scala Spark applications. Entire code is serialized while checkpointing. If code changes, checkpoint data should be truncated.

这篇关于使用Python Spark Direct方法时如何从检查点恢复?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！