这个任务队列设置有什么问题？ [英] Whats wrong with this task queue setup?

问题描述

我在为客户托管的站点上设置了此任务队列实现，它具有每天早晨凌晨2点运行的 cron作业 / admin / tasks / queue ，这会排队发送电子邮件， / admin / tasks / email ，并使用 cursors ，以便以小块进行排队。由于某种原因，昨天晚上 / admin / tasks / queue 继续由此代码运行，因此发送了我的整个电子邮件配额：/ 。我用这段代码做错了什么？

  class QueueUpEmail（webapp.RequestHandler）：
 def post（self）：
 subscribers = Subscriber。 all（）
 subscribers.filter（verified =，True）
 
 last_cursor = memcache.get（'daily_email_cursor'）
如果last_cursor：
 subscribers.with_cursor （last_cursor）
 
 subs = subscribers.fetch（10）
 logging.debug（POST  -  subs count =％i％len（subs））
如果len（subs ）< 10：
 logging.debug（POST  - 少于10个订阅者在子文件中）
＃剩下的订阅者少于10，不要重新计划任务
作为子文件中的
：
 task = taskqueue.Task（url ='/ admin / tasks / email'，params = {'email'：sub.emailaddress，'day'：sub.day_no}）
 task.add（email ）
 memcache.delete（'daily_email_cursor'）
 else：
 logging.debug（POST  - 在subs-reschedule中剩下10个以上子目录）
＃订阅者为10或更重要的是，重新安排
作为子目录：
 task = taskqueue.Task（url ='/ admin / tasks / email'，params = {'email'：sub.emailaddress，'day'：sub。 day_no}）
 task.add（email）
 cursor = subscribers.cursor（）
 memcache.set（'daily_email_cursor'，游标）
 task = taskqueue.Task（ url =/ admin / tasks / queue，params = {}）
 task.add（queueup）
  

解决方案

一些潜在的问题。首先，您将光标存储在内存缓存中，但不保证任何内容。如果在处理过程中发现缓存未命中，您将重新发送每条消息。

其次，如果任务失败，任务将被重新尝试;他们应该被设计成为这个原因的幂等性。当然，在发送电子邮件的情况下，这几乎是不可能的，因为一旦发送了消息，如果您的任务在发送之后由于某种其他原因而死亡，则无法回滚。至少，我建议在发送消息后尝试更新每个Subscriber实体上的最后通过电子邮件发送的日期字段。当然，这本身并不是万无一失的，因为电子邮件发送可能会成功，并且在此之后实体的更新可能会失败。这也会增加整个过程的开销，因为你会为每个订户进行写入。

I've setup this task queue implementation on a site I host for a customer, it has a cron job which runs each morning at 2am "/admin/tasks/queue", this queues up emails to be sent out, "/admin/tasks/email", and uses cursors so as to do the queuing in small chunks. For some reason last night /admin/tasks/queue kept getting run by this code and so sent out my whole quota of emails :/. Have I done something wrong with this code?

class QueueUpEmail(webapp.RequestHandler):
    def post(self):
        subscribers = Subscriber.all()
        subscribers.filter("verified =", True)

        last_cursor = memcache.get('daily_email_cursor')
        if last_cursor:
            subscribers.with_cursor(last_cursor)

        subs = subscribers.fetch(10)
        logging.debug("POST - subs count = %i" % len(subs))
        if len(subs) < 10:
            logging.debug("POST - Less than 10 subscribers in subs")
            # Subscribers left is less than 10, don't reschedule the task
            for sub in subs:
                task = taskqueue.Task(url='/admin/tasks/email', params={'email': sub.emailaddress, 'day': sub.day_no})
                task.add("email")
            memcache.delete('daily_email_cursor')
        else:
            logging.debug("POST - Greater than 10 subscibers left in subs - reschedule")
            # Subscribers is 10 or greater, reschedule
            for sub in subs:
                task = taskqueue.Task(url='/admin/tasks/email', params={'email': sub.emailaddress, 'day': sub.day_no})
                task.add("email")
            cursor = subscribers.cursor()
            memcache.set('daily_email_cursor', cursor)
            task = taskqueue.Task(url="/admin/tasks/queue", params={})
            task.add("queueup")

解决方案

I can see a couple of potential problems. First, you store your cursor in memcache, which is not guaranteed to persist anything. If you get a cache miss halfway through your processing, you'll re-send every message again.

Secondly, tasks will get re-tried if they fail for any reason; they're supposed to be designed to be idempotent for this reason. In the case of sending emails, of course, this is nearly impossible, since once a message is sent it can't be rolled back if your task dies for some other reason after sending it. At a minimum, I'd recommend trying to update a "last emailed date" field on each Subscriber entity after sending them the message. This in itself isn't foolproof, of course, since the email send could succeed and the update of the entity could fail after that. It would also add overhead to the whole process, since you'd be doing a write for each subscriber.

这篇关于这个任务队列设置有什么问题？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！