配置单元 - 可以提取常用选项以便在其他脚本中重用吗？ [英] Hive - Can one extract common options for reuse in other scripts?

问题描述

我有两个Hive脚本，如下所示：

脚本A：

  SET hive.exec.dynamic.partition = true; 
 SET hive.exec.dynamic.partition.mode =非严格; 
 SET hive.exec.parallel = true; 
 
 ...做某事... 
  

脚本B：

  SET hive.exec.dynamic.partition = true; 
 SET hive.exec.dynamic.partition.mode =非严格; 
 SET hive.exec.parallel = true; 
 
 ...做别的... 
  

我们在每个脚本的开始处设置的选项是相同的。是否有可能以某种方式将它们提取出来（例如，在commonoptions.sql中），以便我们的脚本如下所示：

脚本A： p>

 < run commonoptions.sql> 
 
 ...做某事... 
  

脚本B：

 < run commonoptions.sql> 
 
 ...做别的... 
  

理想情况下，我想提取出表格定义，以便我有：

脚本A：

 < run commonoptions.sql> 
<运行defineExternalTableXYZ.sql> 
 ...对表XYZ执行某些操作... 
  

脚本B：

 < run commonoptions.sql> 
<运行defineExternalTableXYZ.sql> 
 ...对表XYZ执行其他操作... 
  

这样我就可以在一个地方管理TableXYZ定义。我没有使用Hive CLI。我将Amazon EMR与Hive Steps结合使用。

解决方案

您可以将这些配置参数存储在通用文件中并加载到每个脚本中使用 source 命令：

  source /tmp/common_init.hql; 
  

您也可以从数据库生成每个工作流程的文件。

I have two Hive scripts which look like this:

Script A:

SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=non-strict;
SET hive.exec.parallel=true;

... do something ...

Script B:

  SET hive.exec.dynamic.partition=true;
  SET hive.exec.dynamic.partition.mode=non-strict;
  SET hive.exec.parallel=true;

... do something else ...

The options that we set at the beginning of each script are the same. Is it possible somehow to extract them out to a common place (for example, into a commonoptions.sql) so that our scripts look like this:

Script A:

 <run commonoptions.sql>

... do something ...

Script B:

 <run commonoptions.sql>

... do something else ...

Ideally I would like to extract out table defintions as well, so that I have:

Script A:

 <run commonoptions.sql>
 <run defineExternalTableXYZ.sql>
... do something with Table XYZ ...

Script B:

 <run commonoptions.sql>
 <run defineExternalTableXYZ.sql>
... do something else with Table XYZ ...

That way I can manage the TableXYZ definition at a single spot. I am not using the Hive CLI. I am using Amazon EMR with Hive Steps.

解决方案

You can store these configuration parameters in common file and load in each of your scripts using source command:

source /tmp/common_init.hql;

Also you can generate this file for each workflow from the database.

这篇关于配置单元 - 可以提取常用选项以便在其他脚本中重用吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！