oozie如何处理依赖关系？ [英] How oozie handle dependencies?

问题描述

我有几个关于oozie 2.3共享库的问题：

目前，我在coordinator.properties中定义了共享库：

  oozie.use.system.libpath = true 
 oozie.libpath =< hdfs_path> 
  

以下是我的问题：

1. 将共享库复制到其他数据节点时，有多少数据节点将获得共享库？

共享库根据协调员作业中的wf数量复制到其他数据节点，还是仅在每个协调员作业中复制一次？ >解决方案向 oozie.libpath 属性中添加条目有效地意味着OOZIE会将这些库配置为当执行工作流中的操作时，mapred.cache.files 配置属性（这是一个DistributedCache属性）。




然后Hadoop会小心每个作业将这些jar复制到每个集群节点一次，然后将这些任务配置到classpath配置属性 mapred.job.classpath.files

中的jar中。



因此，在回答第二个问题时，他们将被复制对于工作流中的每个操作，都不是每个协调器作业一次。因此，如果您有一个具有4个mapreduce操作的wf作业，则该库将在该工作流程的生命周期中被复制到每个任务跟踪器（仅限那些参与mapreduce作业的任务跟踪器）4次。




I have several questions about oozie 2.3 share libraries:

Currently, I defined the share libraries in our coordinator.properties:

oozie.use.system.libpath=true 
oozie.libpath=<hdfs_path>

Here are my questions:

1. When share libraries are copied to other data node and how many data node will get share libraries?

2. Are the share libraries copied to other data node based on number of wf in a coordinator job or they are only copied once per coordinator job?

解决方案

Adding entries to the oozie.libpath property effectively means that OOZIE will configure those libraries to be in the mapred.cache.files configuration property (this is a DistributedCache property) when the actions in your workflow are executed.

Hadoop will then take care of copying those jars to each cluster node once per job, and the tasks are then configured with the jar in the classpath configuration property mapred.job.classpath.files

So in response to your second question, they will be copied over for each action in the workflow, not once per coordinator job. So if you have a wf job that has 4 mapreduce actions, the libraries will be copied to each tasktracker (only those task trackers that participate in the mapreduce job) 4 times in the lifetime of that workflow.

这篇关于oozie如何处理依赖关系？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！