Checkpoint RDD可靠CheckpointRDD具有与原始RDD不同的分区数量 [英] Checkpoint RDD ReliableCheckpointRDD has different number of partitions from original RDD
问题描述
我有两台机器的Spark集群，当我运行Spark Streaming应用程序时，出现以下错误:
I have a spark cluster of two machines and I when I run a spark streaming application I get the following errors :
Exception in thread "main" org.apache.spark.SparkException: Checkpoint RDD ReliableCheckpointRDD[11] at print at StatefulNetworkWordCount.scala:78(1) has different number of partitions from original RDD MapPartitionsRDD[10] at updateStateByKey at StatefulNetworkWordCount.scala:76(2)
at org.apache.spark.rdd.ReliableRDDCheckpointData.doCheckpoint(ReliableRDDCheckpointData.scala:73)
at org.apache.spark.rdd.RDDCheckpointData.checkpoint(RDDCheckpointData.scala:74)
如何在非HDFS/Cassandra/其他任何数据存储的文件系统上提供检查点目录?
How can I give a checkpoint directory on file system that is not HDFS/Cassandra/any other data store ?
我已经想到了两种可能的解决方案，但是我不知道如何对它们进行编码:
I have thought of two possible solutions, but I do not know how to code them :
-
有一个在两个工作人员都本地的远程目录
have one remote directory which is local to both the workers
在两个工作人员上指定一个远程目录
specify a remote directory on to both the workers
有什么建议吗?
推荐答案
好，所以我可以继续选择第一个选项.
Ok, so I was able to go ahead with the first option.
我在所有工作人员上安装了一个远程目录作为检查点，并且运行良好.
I mounted a remote directory on all the workers as checkpoint and it worked perfectly.
How to mount the remote checkpoint directory on the workers:
sudo apt-get install sshfs
Load it to kernel
sudo modprobe fuse
sudo adduser username fuse
mkdir ~/checkpoint
sshfs ubuntu@xx.xx.x.xx:/home/ubuntu/checkpoint ~/checkpoint
这篇关于Checkpoint RDD可靠CheckpointRDD具有与原始RDD不同的分区数量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
