任务数据的位置为NO_PREF.什么时候使用? [英] Task data locality NO_PREF. When is it used?
问题描述
根据Spark文档，数据本地性分为5个级别:
According to Spark doc, there are 5 levels of data locality:
- PROCESS_LOCAL
- NODE_LOCAL
- NO_PREF
- RACK_LOCAL
- 任何
除了NO_PREF以外，所有这些对我来说都很清楚(来自Spark doc:"数据可以从任何地方平等地访问，并且没有位置偏好")
All of them are pretty clear to me apart NO_PREF (from Spark doc: "data is accessed equally quickly from anywhere and has no locality preference")
在什么情况下应该使用NO_PREF?
What is the case NO_PREF whould be used?
推荐答案
RDD特性之一是首选位置.例如，如果RDD源是HDFS文件，则首选位置应包含物理放置数据的数据节点.但是，如果数据来自何处没有区别，或者Spark无法确定首选位置，Spark会在处理此类RDD时创建数据局部性设置为NO_PREF的任务.
One of the RDD characteristics is preferred locations. For example if RDD source is an HDFS file, preferred location should contain data nodes where data is physically located. But if there is no difference where data is coming from or Spark is unable to determine preferred locations, Spark creates tasks with data locality set to NO_PREF during processing such RDDs.
这篇关于任务数据的位置为NO_PREF.什么时候使用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
