如何在Web UI中更改作业/阶段描述? [英] How to change job/stage description in web UI?

问题描述

当我在Apache Spark上运行作业时，Web UI会提供类似于以下的视图:

When I run a job on Apache Spark, the web UI gives a view similar to this:

虽然这对我作为开发人员了解事物的位置非常有用，但我认为阶段描述中的行号对我的支持团队没有太大用处.为了使他们的工作更轻松，我希望能够为我的工作的每个阶段以及工作本身提供一个定制的名称，如下所示:

While this is incredibly useful for me as a developer to see where things are, I think the line numbers in the stage description would be not quite as useful for my support team. To make their job easier, I would like to have the ability to provide a bespoke name for each stage of my job, as well as for the job itself, like so:

这可以在Spark中完成吗?如果是这样，我该怎么办?

Is this something that can be done in Spark? If so, how would I do so?

推荐答案

这是Spark Core非常罕见的功能之一，称为本地属性.

That's where one of the very uncommon features of Spark Core called local properties applies so well.

Spark SQL使用它在单个结构化查询下将不同的Spark作业分组，因此您可以使用SQL选项卡并轻松导航.

Spark SQL uses it to group different Spark jobs under a single structured query so you can use SQL tab and navigate easily.

您可以使用

设置一个本地属性，该属性会影响从该线程提交的作业，例如Spark Fair Scheduler池.也可以在此处设置用户定义的属性.这些属性会传播到工作人员任务，并可以通过org.apache.spark.TaskContext#getLocalProperty访问.

Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool. User-defined properties may also be set here. These properties are propagated through to worker tasks and can be accessed there via org.apache.spark.TaskContext#getLocalProperty.

Web UI使用两个本地属性:

web UI uses two local properties:

• callSite.short在作业"标签中(正是您想要的)
• callSite.long在作业详细信息"页面中.

• callSite.short in Jobs tab (and is exactly what you want)
• callSite.long in Job Details page.

scala> sc.setLocalProperty("callSite.short", "callSite.short")

scala> sc.setLocalProperty("callSite.long", "this is callSite.long")

scala> sc.parallelize(0 to 9).count
res2: Long = 10

以及Web界面中的结果.

And the result in web UI.

单击一项任务以查看详细信息，您可以在其中找到较长的通话站点，即callSite.long.

Click a job to see the details where you can find the longer call site, i.e. callSite.long.

这是阶段"标签.

这篇关于如何在Web UI中更改作业/阶段描述?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！