分布式作业调度，管理和报告 [英] Distributed Job scheduling, management, and reporting

问题描述

我最近和 Hadoop 一起玩，并且对它的日程安排，管理和报告印象深刻MapReduce作业。它似乎使新工作的分配和执行非常顺畅，允许开发人员专注于执行他们的工作。

我想知道Java中是否存在任何东西用于分布式执行不容易表示为MapReduce问题的作业的域？例如：

• 需要任务协调和同步的作业。例如，它们可能涉及按顺序执行任务，但同时执行某些任务是可行的：
  
  

    .-- B  - 。 
   .-- A  -  | |  - 。 
   | ' -  C  - '| 
  开始 -  | |  - 完成
   | | 
  '--D -------------'
    




• 您希望分发CPU密集型任务，但不提供任何输出以减少 - 图像转换/调整大小。

那么是否有一个提供这种分布式计算环境的Java框架/平台？或者，这种事情可以使用Hadoop来接受/实现 - 如果是的话，是否有这些类型的工作的任何模式/指导？我已经找到 Spring Batch 和 Spring Batch Integration ，它似乎满足了我的许多要求。我会让你知道我怎么样。

I recently had a play around with Hadoop and was impressed with it's scheduling, management, and reporting of MapReduce jobs. It appears to make the distribution and execution of new jobs quite seamless, allowing the developer to concentrate on the implementation of their jobs.

I am wondering if anything exists in the Java domain for the distributed execution of jobs that are not easily expressed as MapReduce problems? For example:

• Jobs that require task co-ordination and synchronization. For example, they may involve sequential execution of tasks yet it is feasible to execute some tasks concurrently:

                 .-- B --.
          .--A --|       |--.
          |      '-- C --'  |
  Start --|                 |-- Done
          |                 |
          '--D -------------'
  

• CPU intensive tasks that you'd like to distribute but don't provide any outputs to reduce - image conversion/resizing for example.

So is there a Java framework/platform that provides such a distributed computing environment? Or is this sort of thing acceptable/achievable using Hadoop - and if so are there any patterns/guidelines for these sorts of jobs?

解决方案

I have since found Spring Batch and Spring Batch Integration which appear to address many of my requirements. I will let you know how I get on.

这篇关于分布式作业调度，管理和报告的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！