当 Spark 意识到 RDD 不再被使用时，它会取消持久化 RDD 吗? [英] Would Spark unpersist the RDD itself when it realizes it won't be used anymore?

问题描述

当我们想要多次使用它时，我们可以将 RDD 持久化到内存和/或磁盘中.但是，我们以后是否必须自己取消持久化，或者 Spark 是否会进行某种垃圾收集并在不再需要 RDD 时取消持久化?我注意到如果我自己调用 unpersist 函数，我的性能会变慢.

解决方案

是的，当 RDD 被垃圾回收时，Apache Spark 将取消持久化.

在 RDD.persist 可以看到:

sc.cleaner.foreach(_.registerRDDForCleanup(this))

这会将一个对 RDD 的 WeakReference 放在 ReferenceQueue 中，导致 ContextCleaner.doCleanupRDD 当 RDD 被垃圾回收时.还有:

sc.unpersistRDD(rddId, 阻塞)

有关更多上下文，请参阅 ContextCleaner 和添加它的 commit.>

在依赖垃圾回收处理非持久化 RDD 时需要注意的一些事项:

• RDD 使用执行器上的资源，而垃圾收集发生在驱动器上.在驱动程序有足够的内存压力之前，无论执行程序的磁盘/内存有多满，RDD 都不会自动取消持久化.
• 您不能取消保留 RDD 的一部分(某些分区/记录).如果您从另一个构建一个持久化 RDD，则两者都必须同时完全适合执行程序.

We can persist an RDD into memory and/or disk when we want to use it more than once. However, do we have to unpersist it ourselves later on, or does Spark does some kind of garbage collection and unpersist the RDD when it is no longer needed? I notice that If I call unpersist function myself, I get slower performance.

解决方案

Yes, Apache Spark will unpersist the RDD when it's garbage collected.

In RDD.persist you can see:

sc.cleaner.foreach(_.registerRDDForCleanup(this))

This puts a WeakReference to the RDD in a ReferenceQueue leading to ContextCleaner.doCleanupRDD when the RDD is garbage collected. And there:

sc.unpersistRDD(rddId, blocking)

For more context see ContextCleaner in general and the commit that added it.

A few things to be aware of when relying on garbage collection for unperisting RDDs:

• The RDDs use resources on the executors, and the garbage collection happens on the driver. The RDD will not be automatically unpersisted until there is enough memory pressure on the driver, no matter how full the disk/memory of the executors gets.
• You cannot unpersist part of an RDD (some partitions/records). If you build one persisted RDD from another, both will have to fit entirely on the executors at the same time.

这篇关于当 Spark 意识到 RDD 不再被使用时，它会取消持久化 RDD 吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！