运行Pig脚本时出现堆空间问题 [英] Heap Space Issue while Running a Pig Script

问题描述

我正在尝试执行包含约3000万个数据的Pig脚本，但出现以下堆空间错误:

I am trying to execute a pig script with around 30 million data and I am getting the below heap space error:

> ERROR 2998: Unhandled internal error. Java heap space
> 
> java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Arrays.java:2367)
>         at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
>         at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
>         at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
>         at java.lang.StringBuilder.append(StringBuilder.java:132)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.shiftStringByTabs(LogicalPlanPrinter.java:223)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:108)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:102)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:102)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:102)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:102)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:102)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:102)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:102)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:102)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirstLP(LogicalPlanPrinter.java:83)
>         at org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.visit(LogicalPlanPrinter.java:69)
>         at org.apache.pig.newplan.logical.relational.LogicalPlan.getLogicalPlanString(LogicalPlan.java:148)
>         at org.apache.pig.newplan.logical.relational.LogicalPlan.getSignature(LogicalPlan.java:133)
>         at org.apache.pig.PigServer.execute(PigServer.java:1295)
>         at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
>         at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
>         at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
>         at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
>         at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
>         at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
>         at org.apache.pig.Main.run(Main.java:607)
>         at org.apache.pig.Main.main(Main.java:156)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
> ================================================================================

我用1000万个数据运行了相同的代码，并且运行良好.

I ran the same code with 10 million data and it ran fine.

那么我可以避免上述问题的可能方法有哪些?
压缩是否有助于避免堆空间问题?
我试图将代码拆分为多个片段，但我仍然在 因此，即使我们增加了堆内存分配，它也可以保证，如果我们对volume执行相同的操作，它将保持正确. 数据?

So what are the possible ways I can avoid the above issue?
Does compression helps in avoiding the heap space issue?
I have tried to split the code into multiple fragments and still I am getting the error.So even though if we increase the heap memeory alloaction does it gurantee it will hold true if we execute the same with volume of data?

推荐答案

您可以通过将mapred.map.tasks设置为任意数量来增加映射器的数量.然后运行脚本.

You can increase numbers of mappers by setting mapred.map.tasks to any number you want. and than run your script.

这篇关于运行Pig脚本时出现堆空间问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！