通过键Scalding Hadoop（一个MapReduce作业）写入多个输出 [英] Write to multiple outputs by key Scalding Hadoop, one MapReduce Job

问题描述

如何在单个Map Reduce作业中使用Scalding（/ cascading）来写入多个依赖于键的输出。我当然可以为所有可能的键使用 .filter ，但这是一个可怕的黑客攻击，它会触发许多作业。

How can you write to multiple outputs dependent on the key using Scalding(/cascading) in a single Map Reduce Job. I could of course use .filter for all the possible keys, but that is a horrible hack, which will fire up many jobs.

推荐答案

TemplatedTsv Scalding（从版本0.9.0rc16开始），与Cascading TemplateTsv完全一样。

There is TemplatedTsv in Scalding (from version 0.9.0rc16 and up), exactly same as Cascading TemplateTsv.

Tsv(args("input"), ('COUNTRY, 'GDP))
.read
.write(TemplatedTsv(args("output"), "%s", 'COUNTRY))
// it will create a directory for each country under "output" path in Hadoop mode.

这篇关于通过键Scalding Hadoop（一个MapReduce作业）写入多个输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！