诊断Boto3中的内存泄漏 [英] Diagnosing Memory leak in boto3

问题描述

我有一个在Elastic Beanstalk上运行的芹菜工人，该工人轮询SQS队列，获取消息(包含S3文件名)，从S3下载这些文件并进行处理.我的工作人员计划每15秒运行一次，但是由于某种原因，内存使用量会随着时间的推移而不断增加.

I have a celery worker running on Elastic Beanstalk that polls a SQS queue, gets messages (containing S3 file names), downloads those files from S3 and processes them. My worker is scheduled to run at every 15 seconds but due to some reason the memory usage keeps on increasing with time.

这是我用来访问SQS的代码

This is the code I'm using to access SQS

def get_messages_from_sqs(queue_url, queue_region="us-west-2", number_of_messages=1):
    client = boto3.client('sqs', region_name=queue_region)
    sqs_response = client.receive_message(QueueUrl=queue_url, MaxNumberOfMessages=number_of_messages)
    messages = sqs_response.get("Messages", [])
    cleaned_messages = []
    for message in messages:
        body = json.loads(message["Body"])
        data = body["Records"][0]
        data["receipt_handle"] = message["ReceiptHandle"]
        cleaned_messages.append(data)
    return cleaned_messages

def download_file_from_s3(bucket_name, filename):
    s3_client = boto3.client('s3')
    s3_client.download_file(bucket_name, filename, '/tmp/{}'.format(filename))

在完成该过程之后，是否需要在boto3中关闭客户端连接?如果是这样，我们该怎么做?

Do we need to close client connection in boto3 after we're done with the process ? If so, how can we do it ?

推荐答案

我在生产中使用Celery遇到了类似的问题，与Boto完全无关.尽管我没有内存泄漏的解释(这将花费一些严重的代码进行分析和分析)，但是如果您的目标只是不耗尽内存，我可以提供一种潜在的解决方法.

I have run into similar issues using Celery in production, completely unrelated to Boto. Although I do not have an explanation for the memory leak (this would take some serious code spelunking and profiling), I can offer a potential workaround if your goal is just to not run out of memory.

设置每个孩子的最大任务数应该允许您不断回收被终止进程释放的内存.

Setting max tasks per child should allow you to constantly reclaim the memory as it is released by the killed process.

这篇关于诊断Boto3中的内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！