如何在AWS Lambda中增加Java堆大小? [英] How to increase Java Heap size in aws Lambda?

问题描述

我正在使用AWS开发工具包 http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/package-summary.html

在AWS Lambda中对数据处理进行编程

遇到错误

    {
  "errorMessage": "Java heap space",
  "errorType": "java.lang.OutOfMemoryError",
  "stackTrace": [
    "java.util.Arrays.copyOf(Arrays.java:3332)",
    "java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)",
    "java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)",
    "java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)",
    "java.lang.StringBuffer.append(StringBuffer.java:369)",
    "java.io.StringWriter.write(StringWriter.java:94)",
    "org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1129)",
    "org.apache.commons.io.IOUtils.copy(IOUtils.java:1104)",
    "org.apache.commons.io.IOUtils.copy(IOUtils.java:1050)",
    "org.apache.commons.io.IOUtils.copy(IOUtils.java:1075)",
    "org.apache.commons.io.IOUtils.toString(IOUtils.java:382)",
    "example.Hello.myHandler(Hello.java:47)",
    "sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)",
    "sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)",
    "sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)",
    "java.lang.reflect.Method.invoke(Method.java:497)"
  ]
}

我想通过编程方式增加堆大小来解决此问题

导致此问题的代码是

AmazonS3 s3Client = new AmazonS3Client();
S3Object s3Object = s3Client.getObject(new GetObjectRequest(srcBucket, srcKey));
InputStream objectData = s3Object.getObjectContent();

// transfer data into JSON
String theString = IOUtils.toString(objectData, null); 

我提取的对象约为300MB.的配置和配置

Memory: 1024 MB

解决方案

使用 Gson 或 Jackson API来解析您的Json对象，而不是使用Lambda限制./p>

类似的东西可以帮助您开始

AmazonS3 s3Client = new AmazonS3Client();
S3Object s3Object = s3Client.getObject(new GetObjectRequest(srcBucket, srcKey));
InputStream objectData = s3Object.getObjectContent();

// transfer data into JSON
JsonFactory jsonfactory = new JsonFactory(); 
JsonParser parser = jsonfactory.createJsonParser(objectData); 

// starting parsing of JSON String 
while (parser.nextToken() != JsonToken.END_OBJECT) { 
    // do what you need to do
}

I am using AWS SDK http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/package-summary.html

to program a data processing in AWS Lambda

encountered the error

    {
  "errorMessage": "Java heap space",
  "errorType": "java.lang.OutOfMemoryError",
  "stackTrace": [
    "java.util.Arrays.copyOf(Arrays.java:3332)",
    "java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)",
    "java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)",
    "java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)",
    "java.lang.StringBuffer.append(StringBuffer.java:369)",
    "java.io.StringWriter.write(StringWriter.java:94)",
    "org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1129)",
    "org.apache.commons.io.IOUtils.copy(IOUtils.java:1104)",
    "org.apache.commons.io.IOUtils.copy(IOUtils.java:1050)",
    "org.apache.commons.io.IOUtils.copy(IOUtils.java:1075)",
    "org.apache.commons.io.IOUtils.toString(IOUtils.java:382)",
    "example.Hello.myHandler(Hello.java:47)",
    "sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)",
    "sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)",
    "sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)",
    "java.lang.reflect.Method.invoke(Method.java:497)"
  ]
}

I want to solve this by increase heap size programmatically

The code caused this problem is

AmazonS3 s3Client = new AmazonS3Client();
S3Object s3Object = s3Client.getObject(new GetObjectRequest(srcBucket, srcKey));
InputStream objectData = s3Object.getObjectContent();

// transfer data into JSON
String theString = IOUtils.toString(objectData, null); 

The object I am extracting is about 300MB. and Configuration of Lambda

Memory: 1024 MB

解决方案

You will be better off using Gson or Jackson APIs to parse your Json object rather than working with Lambda limitation.

something like that can help you started

AmazonS3 s3Client = new AmazonS3Client();
S3Object s3Object = s3Client.getObject(new GetObjectRequest(srcBucket, srcKey));
InputStream objectData = s3Object.getObjectContent();

// transfer data into JSON
JsonFactory jsonfactory = new JsonFactory(); 
JsonParser parser = jsonfactory.createJsonParser(objectData); 

// starting parsing of JSON String 
while (parser.nextToken() != JsonToken.END_OBJECT) { 
    // do what you need to do
}

这篇关于如何在AWS Lambda中增加Java堆大小?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！