HDFS中查找的行总数使用命令行分布式文件 [英] Finding total number of lines in hdfs distributed file using command line

问题描述

我的工作，其中一个数据是保存在 HDFS 以分散的方式在群集上。以下是我有：

I am working on a cluster where a dataset is kept in hdfs in distributed manner. Here is what I have:

[hmi@bdadev-5 ~]$ hadoop fs -ls /bdatest/clm/data/
Found 1840 items
-rw-r--r--   3 bda supergroup          0 2015-08-11 00:32 /bdatest/clm/data/_SUCCESS
-rw-r--r--   3 bda supergroup   34404390 2015-08-11 00:32 /bdatest/clm/data/part-00000
-rw-r--r--   3 bda supergroup   34404062 2015-08-11 00:32 /bdatest/clm/data/part-00001
-rw-r--r--   3 bda supergroup   34404259 2015-08-11 00:32 /bdatest/clm/data/part-00002
....
....

数据的格式为：

[hmi@bdadev-5 ~]$ hadoop fs -cat /bdatest/clm/data/part-00000|head
V|485715986|1|8ca217a3d75d8236|Y|Y|Y|Y/1X||Trimode|SAMSUNG|1x/Trimode|High|Phone|N|Y|Y|Y|N|Basic|Basic|Basic|Basic|N|N|N|N|Y|N|Basic-Communicator|Y|Basic|N|Y|1X|Basic|1X|||SAM|Other|SCH-A870|SCH-A870|N|N|M2MC|

所以，我想要做的是计算在原始数据文件数据行的总数。我的理解是，像分布式块部分00000 ，部分-00001 等有重叠。所以，只是计算的行数在部分-XXXX 文件，总结他们将无法正常工作。另外，原始数据集数据是大小〜70GB 的。我怎样才能有效地找出线路总数？

So, what I want to do is to count the total number of lines in the original data file data. My understanding is that the distributed chunks like part-00000, part-00001 etc have overlaps. So just counting the number of lines in part-xxxx files and summing them won't work. Also the original dataset data is of size ~70GB. How can I efficiently find out the total number of lines?

推荐答案

更​​多effieciently - 您可以使用火花计数没有。行。下面code片段有助于计数的行数。

More effieciently -- you can use spark to count the no. of lines. The following code snippet helps to count the number of lines.

text_file = spark.textFile("hdfs://...")
count = text_file.count();
count.dump();

这显示没有计数。行。

注：在不同的部分文件中的数据不会重叠

使用 HDFS DFS -cat / bdatest / CLM /数据/兼职* |厕所-l 也会给你的输出，但这将所有的数据转储到本地机器需要较长的时间。

Using hdfs dfs -cat /bdatest/clm/data/part-* | wc -l will also give you the output but this will dump all the data to the local machine and takes longer time.

最好的解决办法是使用MA preduce或火花。马predue将需要更长的时间来制定和执行。如果安装了sprak，这是最好的选择。

Best solution is to use mapreduce or spark. Mapredue will take longer time to develop and execute. If the sprak is installed, this is the best choice.

这篇关于HDFS中查找的行总数使用命令行分布式文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！