在Lambda中创建AmazonS3Client时出现OutOfMemoryError [英] OutOfMemoryError when creating AmazonS3Client in Lambda

查看：176
本文介绍了在Lambda中创建AmazonS3Client时出现OutOfMemoryError的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我有一个AWS Lambda函数，该函数仅配置有128MB的内存，由SNS触发(它本身由S3触发)，并将从S3下载文件.

I have an AWS Lambda function, configured with only 128MB of memory, is triggered by SNS (which is itself triggered by S3) and will download the file from S3.

在我的职务上，我有以下内容:

In my function, I have the following:

public class LambdaHandler {

    private final AmazonS3Client s3Client = new AmazonS3Client();

    public void gdeltHandler(SNSEvent event, Context context) {
        System.out.println("Starting");
        System.out.println("Found " + eventFiles.size() + " event files");
    }

我已经注释掉了所有逻辑，并从这篇文章中排除了所有逻辑，因为我遇到了OutOfMemoryError，该错误与创建AmazonS3Client对象无关.当我拿出那个物体时，我没有得到错误.上面的确切代码会导致OutOfMemoryError.

I've commented out and excluded from this post all of the logic because I am getting an OutOfMemoryError which I have isolated to the creation of the AmazonS3Client object. When I take that object out, I don't get the error. The exact above code results in the OutOfMemoryError.

我为该功能分配了128MB的内存，这真的不足以简单地获取凭证并实例化AmazonS3Client对象吗?

I assigned 128MB of memory to the function, is that really not enough to simply grab the credentials and instantiate the AmazonS3Client object?

我尝试提供AmazonS3Client构造函数

I've tried giving the AmazonS3Client constructor

new EnvironmentVariableCredentialsProvider()

以及

new InstanceProfileCredentialsProvider()

具有相似的结果.

创建AmazonS3Client对象是否仅需要更多内存?

Does the creation of the AmazonS3Client object simply require more memory?

下面是堆栈跟踪:

元空间:java.lang.OutOfMemoryError java.lang.OutOfMemoryError: 的元空间 com.fasterxml.jackson.databind.deser.BeanDeserializerBuilder.build(BeanDeserializerBuilder.java:347) 在 com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:242) 在 com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143) 在 com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:409) 在 com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:358) 在 com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:265) 在 com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:245) 在 com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:143) 在 com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:439) 在 com.fasterxml.jackson.databind.ObjectReader._prefetchRootDeserializer(ObjectReader.java:1588) 在 com.fasterxml.jackson.databind.ObjectReader.(ObjectReader.java:185) 在 com.fasterxml.jackson.databind.ObjectMapper._newReader(ObjectMapper.java:558) 在 com.fasterxml.jackson.databind.ObjectMapper.reader(ObjectMapper.java:3108)

Metaspace: java.lang.OutOfMemoryError java.lang.OutOfMemoryError: Metaspace at com.fasterxml.jackson.databind.deser.BeanDeserializerBuilder.build(BeanDeserializerBuilder.java:347) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:242) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143) at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:409) at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:358) at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:265) at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:245) at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:143) at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:439) at com.fasterxml.jackson.databind.ObjectReader._prefetchRootDeserializer(ObjectReader.java:1588) at com.fasterxml.jackson.databind.ObjectReader.(ObjectReader.java:185) at com.fasterxml.jackson.databind.ObjectMapper._newReader(ObjectMapper.java:558) at com.fasterxml.jackson.databind.ObjectMapper.reader(ObjectMapper.java:3108)

当我尝试提供InstanceProfileCredentialsProvider或EnvironmentVariableCredentialsProvider时，得到以下堆栈跟踪:

When I try providing the InstanceProfileCredentialsProvider or EnvironmentVariableCredentialsProvider, I get the following stack trace:

线程"main"中的异常java.lang.Error: java.lang.OutOfMemoryError:元空间位于 lambdainternal.AWSLambda.(AWSLambda.java:62)在 java.lang.Class.forName0(本机方法)，位于 java.lang.Class.forName(Class.java:348)在 lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94)由以下原因引起: java.lang.OutOfMemoryError:元空间位于 java.lang.ClassLoader.defineClass1(本机方法)位于 java.lang.ClassLoader.defineClass(ClassLoader.java:763)在 java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) 在java.net.URLClassLoader.defineClass(URLClassLoader.java:467)在 java.net.URLClassLoader.access $ 100(URLClassLoader.java:73)在 java.net.URLClassLoader $ 1.run(URLClassLoader.java:368)在 java.net.URLClassLoader $ 1.run(URLClassLoader.java:362)在 java.security.AccessController.doPrivileged(本机方法)，位于 java.net.URLClassLoader.findClass(URLClassLoader.java:361)在 java.lang.ClassLoader.loadClass(ClassLoader.java:424)在 java.lang.ClassLoader.loadClass(ClassLoader.java:357)在 lambdainternal.EventHandlerLoader $ PojoMethodRequestHandler.makeRequestHandler(EventHandlerLoader.java:421) 在 lambdainternal.EventHandlerLoader.getTwoLengthHandler(EventHandlerLoader.java:777) 在 lambdainternal.EventHandlerLoader.getHandlerFromOverload(EventHandlerLoader.java:802) 在 lambdainternal.EventHandlerLoader.loadEventPojoHandler(EventHandlerLoader.java:888) 在 lambdainternal.EventHandlerLoader.loadEventHandler(EventHandlerLoader.java:740) 在 lambdainternal.AWSLambda.findUserMethodsImmediate(AWSLambda.java:126) 在lambdainternal.AWSLambda.findUserMethods(AWSLambda.java:71)在 lambdainternal.AWSLambda.startRuntime(AWSLambda.java:219)在 lambdainternal.AWSLambda.(AWSLambda.java:60)...还有3个开始 RequestId:58837136-483e-11e6-9ed3-39246839616a版本:$ LATEST END RequestId:58837136-483e-11e6-9ed3-39246839616a REPORT RequestId: 58837136-483e-11e6-9ed3-39246839616a持续时间:15002.92 ms已结算 持续时间:15000 ms内存大小:128 MB已使用的最大内存:50 MB
2016-07-12T14:40:28.048Z 58837136-483e-11e6-9ed3-39246839616a任务 15.00秒后超时

Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Metaspace at lambdainternal.AWSLambda.(AWSLambda.java:62) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94) Caused by: java.lang.OutOfMemoryError: Metaspace at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.makeRequestHandler(EventHandlerLoader.java:421) at lambdainternal.EventHandlerLoader.getTwoLengthHandler(EventHandlerLoader.java:777) at lambdainternal.EventHandlerLoader.getHandlerFromOverload(EventHandlerLoader.java:802) at lambdainternal.EventHandlerLoader.loadEventPojoHandler(EventHandlerLoader.java:888) at lambdainternal.EventHandlerLoader.loadEventHandler(EventHandlerLoader.java:740) at lambdainternal.AWSLambda.findUserMethodsImmediate(AWSLambda.java:126) at lambdainternal.AWSLambda.findUserMethods(AWSLambda.java:71) at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:219) at lambdainternal.AWSLambda.(AWSLambda.java:60) ... 3 more START RequestId: 58837136-483e-11e6-9ed3-39246839616a Version: $LATEST END RequestId: 58837136-483e-11e6-9ed3-39246839616a REPORT RequestId: 58837136-483e-11e6-9ed3-39246839616a Duration: 15002.92 ms Billed Duration: 15000 ms Memory Size: 128 MB Max Memory Used: 50 MB
2016-07-12T14:40:28.048Z 58837136-483e-11e6-9ed3-39246839616a Task timed out after 15.00 seconds

编辑1 如果我将分配给该功能的内存增加到192MB，它就可以正常工作，尽管这很奇怪，但报告仅使用cloudwatch日志中的59MB内存.我只是失去了其余的记忆吗?

EDIT 1 If I increase the memory allocated to the function to even 192MB, it works just fine, though strangely enough, reports only using 59MB of memory in the cloudwatch logs. Am I simply losing the rest of the memory?

推荐答案

在Lambda函数中使用AWS Java SDK时，我一直在观察这一点. 创建任何AWS客户端(同步或异步)时，您似乎都可能会脱离元空间.

I have been observing this when using AWS Java SDK within the Lambda function. It would seem when creating any of the AWS clients (Sync or Async) you may get out of Metaspace.

我认为这是由于Amazon Client在实例化时执行的事情所致，包括AmazonHttpClient的创建以及请求处理程序链的动态加载(AmazonEc2Client#init() private方法的一部分).

I believe this is due to things that the Amazon Client is performing upon instantiation, including AmazonHttpClient creation as well as dynamic loading of request handler chains (part of AmazonEc2Client#init() private method).

报告的内存使用情况可能是针对堆本身的，但可能不包括元空间. AWS论坛上有几个主题，但AWS对此没有任何回应.

It is possible that the reported memory usage is for Heap itself, but may not include Metaspace. There are a few threads on AWS Forums but no responses from AWS on the matter.

这篇关于在Lambda中创建AmazonS3Client时出现OutOfMemoryError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆