Celery + gevent仅使用一个CPU内核 [英] Celery + gevent using only one CPU core
问题描述
我在用gevent运行Celery时遇到了性能负载问题,一切都在我的VPS的同一内核上运行.
I've issues with performance load running Celery with gevent, everything is running on the same core on my VPS.
这里是4个Celery实例的屏幕快照,每个实例具有20个gevent并发性.
Here's a screenshot of 4 Celery instance with 20 gevent concurrency each
如何解决此问题?我在做什么错了?
How to fix this ? What am I doing wrong ?
这是我的第一个任务:
def update_sender():
items = models.Item.objects.filter(active=True).all()
count = items.count()
items = [i.id for i in items]
step = count / settings.WORKERS
for job in list(chunks(items, step)):
update_item.apply_async(args=[job])
调用以下子任务:
def update_item(items):
for item in items:
try:
i = models.Item.objects.get(id=item)
url = "someurl"
rep = requests.get(url)
jrep = rep.json()
tracker = ItemTracker(i, jrep)
if tracker.skip():
continue
if tracker.method1():
if not tracker.method2():
tracker.method3()
tracker.save()
这都是关于执行大量HTTP请求并同时更新数据库的事情.
It's all about doing a lot of HTTP requests and updating database concurrently.
推荐答案
带有gevent的Celery仍仅使用一个进程,它只是在进程内部启动多个greenlet,但仍一次仅执行一个greenlet.要允许使用多个内核,您需要使用芹菜多
Celery with gevent still only uses a single process, its just starting multiple greenlets inside of the process, but it is still only executing one greenlet at a time. To allow using more than 1 core, you need to start multiple celery processes using something like celery-multi
这篇关于Celery + gevent仅使用一个CPU内核的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!