在Java中的Spark中获取当前任务ID [英] Get current task ID in Spark in Java

问题描述

我需要获取Spark中当前任务的ID.我一直在Google和官方API中进行搜索，但是我只能找到执行者ID和RDD的ID.有谁知道如何获取任务的唯一ID?我已经看到 TaskInfo 类完全符合我的要求，但是我不知道如何获取该类的实例.

I need to get the ID of the current task in Spark. I have been searching in Google and in the official API but the only IDs I can find are the executor ID and the ID of the RDD. Does anyone know how to get the unique ID of a task? I have seen that the class TaskInfo has exactly what I am looking for, but I do not know how to get an instance of this class.

推荐答案

为了获取特定的任务ID，您可以使用

In order to get the specific task ID you can use the TaskContext:

import org.apache.spark.TaskContext;

textFile.map( x -> {
    TaskContext tc = TaskContext.get();
    System.out.println(tc.taskAttemptId());
});

请记住，特定的println将打印在当前执行的节点上，而不是驱动程序控制台上.

Bear in mind that the specific println will be printed on the node it is currently executed and not the drivers console.

这篇关于在Java中的Spark中获取当前任务ID的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！