更改Hadoop上随机数据生成的大小 [英] Change the size of random data generation on Hadoop

问题描述

我正在使用 RandomWriter 函数在Hadoop上运行排序示例.此特定功能使用Map/Reduce将10 gig(默认情况下)的随机数据/主机写入DFS.

I am running sort example on Hadoop using RandomWriter function. This particular function writes 10 gig (by default) of random data/host to DFS using Map/Reduce.

bin/hadoop jar hadoop-*-examples.jar randomwriter <out-dir>.

任何人都可以告诉我如何更改 RandomWriter 函数的10GB大小吗?

Can anyone please tell how can I change the size 10GB of RandomWriter function?

推荐答案

该示例具有一些可配置的参数.这些参数在配置文件中提供给jar.运行时将其用作(提供配置文件)

That example have some configurable parameters. These parameters are given to jar in a config file. To run use it as (suppling a config file)

bin/hadoop jar hadoop-*-examples.jar randomwriter <out-dir> [<configuration file>]

或使用以下参数运行它

bin/hadoop jar hadoop-*-examples.jar randomwriter 
 -Dtest.randomwrite.bytes_per_map=<value> 
 -Dtest.randomwriter.maps_per_host=<value> <out-dir> [<configuration file>]

有关所有可配置参数的详细信息，请参见: https://wiki.apache.org/hadoop/RandomWriter

For details about all configurable parameters see : https://wiki.apache.org/hadoop/RandomWriter

这篇关于更改Hadoop上随机数据生成的大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！