Hive中的字数统计程序 [英] Word Count program in Hive

问题描述

我正在学习Hive。令人惊讶的是，我找不到如何编写简单的字数统计工作的例子。是否正确？

假设我有一个输入文件 input.tsv ：

  hello，world 
这是一个输入文件示例
  

我在Python中创建了一个分隔符，将每一行转换为单词：

  import sys $ b.b 
为sys.stdin中的行：
为line.split（）中的单词：
打印字
  

然后在我的Hive脚本中有以下内容：

  CREATE TABLE input （line STRING）; 
 LOAD DATA LOCAL INPATH'input.tsv'OVERWRITE INTO TABLE input; 
 
  - 用于存放单词的临时表... 
 CREATE TABLE单词（单词STRING）; 
 
添加文件splitter.py; 
 
 INSERT OVERWRITE TABLE单词
 SELECT TRANSFORM（文本）
使用'python splitter.py'
 AS字
 FROM input; 
 
 SELECT word，count（*）AS FROM FROM GROUP GROUP BY word; 
  

我不确定我是否错过了某些东西，或者确实如此复杂。 （特别是，我是否需要临时的 words 表，并且是否需要编写外部分隔符函数？）

  SELECT word，COUNT 
 （*）FROM输入LATERAL VIEW explode（split（text，''））lTable as word GROUP BY word; 
  

我使用横向视图来启用表值函数（explode）来自分割函数并为每个值输出一个新行。在实践中，我使用UDF来封装IBM的ICU4J分词器。我通常不使用转换脚本，并使用UDF来处理所有事情。你不需要临时单词表。

I'm trying to learn Hive. Surprisingly, I can't find an example of how to write a simple word count job. Is the following correct?

Let's say I have an input file input.tsv:

hello, world
this is an example input file

I create a splitter in Python to turn each line into words:

import sys

for line in sys.stdin:
 for word in line.split():
   print word

And then I have the following in my Hive script:

CREATE TABLE input (line STRING);
LOAD DATA LOCAL INPATH 'input.tsv' OVERWRITE INTO TABLE input;

-- temporary table to hold words...
CREATE TABLE words (word STRING);

add file splitter.py;

INSERT OVERWRITE TABLE words 
  SELECT TRANSFORM(text) 
    USING 'python splitter.py' 
    AS word
  FROM input;

SELECT word, count(*) AS count FROM words GROUP BY word;

I'm not sure if I'm missing something, or if it really is this complicated. (In particular, do I need the temporary words table, and do I need to write the external splitter function?)

解决方案

If you want a simple one see the following:

SELECT word, COUNT(*) FROM input LATERAL VIEW explode(split(text, ' ')) lTable as word GROUP BY word;

I use a lateral view to enable the use of a table valued function (explode) which takes the list that comes out of split function and outputs a new row for every value. In practice I use a UDF that wraps IBM's ICU4J word breaker. I generally don't use transform scripts and use UDFs for everything. You don't need a temporary words table.

这篇关于Hive中的字数统计程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！