使用python spark直接方法时如何从检查点恢复? [英] How to recover from checkpoint when using python spark direct approach?
问题描述
阅读官方文档后,我尝试使用checkpoint
和 getOrCreate
在火花流中.一些片段:
After read official docs, i tried using checkpoint
with getOrCreate
in spark streaming. Some snippets:
def get_ssc():
sc = SparkContext("yarn-client")
ssc = StreamingContext(sc, 10) # calc every 10s
ks = KafkaUtils.createDirectStream(
ssc, ['lucky-track'], {"metadata.broker.list": KAFKA_BROKER})
process_data(ks)
ssc.checkpoint(CHECKPOINT_DIR)
return ssc
if __name__ == '__main__':
ssc = StreamingContext.getOrCreate(CHECKPOINT_DIR, get_ssc)
ssc.start()
ssc.awaitTermination()
代码可以很好地用于恢复,但恢复的上下文始终适用于旧的进程函数.这意味着即使我更改了 map/reduce 函数代码,它也根本不起作用.
The code works fine for recover, but the recovered context always works on the old process function. It means that even if i changed map/reduce function code, it not works at all.
直到现在,spark(1.5.2) 仍然不支持 python 的任意偏移量.那么,我应该怎么做才能使其正常工作?
Until now, spark(1.5.2) still not support arbitrary offset for python. So, what should i do to make this work properly?
推荐答案
这种行为是设计使然",并且对 java/scala Spark 应用程序也有效.整个代码在检查点时被序列化.如果代码发生变化,检查点数据应该被截断.
Such behaviour is "by design", and is valid also for java/scala Spark applications. Entire code is serialized while checkpointing. If code changes, checkpoint data should be truncated.
这篇关于使用python spark直接方法时如何从检查点恢复?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!