在同一应用程序的任务（映射函数）中创建Spark任务 [英] Creating spark tasks from within tasks (map functions) on the same application
问题描述
是否可以从pyspark中的映射器功能（即从任务）进行映射？
换句话说，是否可以从任务中打开子任务？
如果是这样-我如何将sparkContext传递给任务-就像一个变量一样？
Is it possible to do a map from a mapper function (i.e from tasks) in pyspark? In other words, is it possible to open "sub tasks" from a task? If so - how do i pass the sparkContext to the tasks - just as a variable?
我想要一份由许多任务组成的工作-这些任务中的每一个也应该创建许多任务，而无需回到驱动程序。
I would like to have a job that is composed from many tasks - each of these tasks should create many tasks as well, without going back to the driver.
我的用例是这样的：
我正在编写代码将使用工作队列编写的应用程序移植到pyspark。
在我的旧应用程序中，任务创建了其他任务-我们使用了此功能。我不想重新设计整个代码，因为此举引发了火花（特别是因为我将必须确保两个平台都在系统之间的过渡阶段工作）...
My use case is like this: I am doing a code porting of an application that was written using work queues - to pyspark. In my old application tasks created other tasks - and we used this functionality. I don't want to redesign the whole code because of the move to spark (especially because i will have to make sure that both platform works in the transient phase between the systems)...
推荐答案
是否可以从任务中打开子任务？
Is it possible to open "sub tasks" from a task?
否，至少不是以健康的方式 * 。
No, at least not in a healthy manner*.
A task 是从驱动程序发送的命令，Spark具有 一个驱动程序 （中央协调器），可以与许多分布式
A task is a command sent from the driver and Spark has as one Driver (central coordinator) that communicates with many distributed workers (executors).
因此，您在这里的要求意味着每个任务都可以扮演次级驱动者的角色。甚至没有一个工人，对我的回答与任务有着相同的信念。
As a result, what you ask for here, implies that every task can play the role of a sub-Driver. Not even a worker, which would have the same faith in my answer as the task.
卓越的资源：
- What is a task in Spark? How does the Spark worker execute the jar file?
- What are workers, executors, cores in Spark Standalone cluster?
？Spark Standalone集群中的工作人员，执行者，核心是什么？
* 话虽如此，我的意思是我不知道有任何骇客之类的东西，如果存在的话可能太具体了。
这篇关于在同一应用程序的任务（映射函数）中创建Spark任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
