即使作业成功完成，内存使用率也不会降低 [英] Memory usage not getting lowered even after job is completed successfully

问题描述

我在 apscheduler 中添加了一个作业，它在内存中加载了一些数据，并且在作业完成后我将删除所有对象.现在，如果我使用 python 运行此作业，它会成功运行，并且进程成功退出后内存会下降.但在 apscheduler 的情况下，内存使用量不会下降.我正在使用 BackgroundScheduler.提前致谢.

I have a job added in apscheduler which loads some data in memory and I am deleting all the objects after the job is complete. Now if I run this job with python it works successfully and memory drop after process exits successfully.But in case of apscheduler the memory usage is not coming down.I am using BackgroundScheduler.Thanks in advance.

推荐答案

我通过 apscheduler 运行了不少任务.我怀疑这个设置会导致 Heroku 上的 R14 错误，每天都会发生 dyno 内存过载、崩溃和重启.因此，我启动了另一个 dyno 并安排了一些作业以非常频繁地运行.

I was running quite a few tasks via apscheduler. I suspected this setup led to R14 errors on Heroku, with dyno memory overload, crashes and restarts occurring daily. So I spun up another dyno and scheduled a few jobs to run very frequently.

查看 Heroku 中的指标选项卡，立即发现 apscheduler 是罪魁祸首.

Watching the metrics tab in Heroku, it immediately became clear that apscheduler was the culprit.

建议在作业运行后删除对我来说.但这在运行 cron 和间隔作业时当然是个坏主意，因为它们不会再次运行.

Removing jobs after they're run was recommended to me. But this is of course a bad idea when running cron and interval jobs as they won't run again.

最终解决的是调整线程池执行器(降低最大工人数)，在 Stackoverflow 上看到这个答案 和 this 和 <一个 href="https://github.com/agronholm/apscheduler/issues/235" rel="nofollow noreferrer">this 在 Github 上的帖子.我绝对建议您阅读相关文档.

What finally solved it was tweaking the threadpoolexecutioner (lowering max number of workers), see this answer on Stackoverflow and this and this post on Github. I definitely suggest you read the docs on this.

其他诊断资源:1, 2.

示例代码:

import logging
from apscheduler.executors.pool import ThreadPoolExecutor, ProcessPoolExecutor
from apscheduler.schedulers.blocking import BlockingScheduler
from tests import overloadcheck

logging.basicConfig()
logging.getLogger('apscheduler').setLevel(logging.DEBUG)

sched = BlockingScheduler(
    executors={
        'threadpool': ThreadPoolExecutor(max_workers=9),
        'processpool': ProcessPoolExecutor(max_workers=3)
        }
)

@sched.scheduled_job('interval', minutes=10, executor='threadpool')
def message_overloadcheck():
    overloadcheck()

sched.start()

或者，如果您喜欢我，喜欢运行繁重的任务 — 尝试将 ProcessPoolExecutor 作为替代方案，或作为 ThreadPool 的补充，但请确保在这种情况下从特定作业调用它.

Or, if you like I do, love to run heavy tasks—try the ProcessPoolExecutor as an alternative, or addition to the ThreadPool, but make sure to call it from specific jobs in such case.

更新:而且，如果您想使用它，还需要导入 ProcessPoolExecutor，将其添加到代码中.

Update: And, you need to import ProcessPoolExecutor as well if you wish to use it, added this to code.

这篇关于即使作业成功完成，内存使用率也不会降低的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！