合并cron作业以减少实例数量 [英] Combine cron jobs to reduce number of instances

问题描述

假设我有3个请求:

@app.route('/schedulejob1/')
def schedulejob1():
   for pram in prams1:
     deferred.defer(scheduletjob1Task,pram)
   return 'Running'
@app.route('/schedulejob1/')
def schedulejob1():
   for pram in prams2:
     deferred.defer(scheduletjob1Task,pram)
   return 'Running'
@app.route('/schedulejob1/')
def schedulejob1():
   for pram in prams3:
     deferred.defer(scheduletjob1Task,pram)
   return 'Running'

所以我的cron.yaml是

So my cron.yaml is

cron:
- description: Run schedule job1
  url: /schedulejob1/
  schedule: every 1 minutes
- description: Run schedule job2
  url: /schedulejob2/
  schedule: every 1 minutes
- description: Run schedule job3
  url: /schedulejob3/
  schedule: every 1 minutes

此方法每分钟将有3个请求，因此gae将有很大机会动态创建更多实例，从而增加了成本(实例x前端/后端小时).

This approach will have 3 requests per minutes so that there is a big chance gae will dynamically create more instances which increase cost of (instances x frondend/backend hours).

我的计划是将3个请求放到一个请求中:

My plan is to put 3 requests into a single request:

@app.route('/schedulejobs/')
def schedulejobs():
   for pram in prams1
     deferred.defer(scheduletjob1Task,pram)
   for pram in prams2  
     deferred.defer(scheduletjob2Task,pram)
   for pram in prams3 
     deferred.defer(scheduletjob3Task,pram)

然后

 cron:
    - description: Run schedule jobs
      url: /schedulejobs/

第二种方法是否有助于减少请求数量?因此降低成本?

Does the second approach help to reduce the number of requests? therefore reduce cost?

我的目标是将请求分成多个任务队列，每个任务队列在不到1分钟的时间内运行，并且可能永久失败(或可以重试1次). 我还将在后端实例上放置一半的cron作业，以利用9个小时的空闲时间.

My objective is to break a request into multiple task queues, each taskqueue run in less than 1 minutes, and it can fail permanently (or can retry 1 time). I also will put a a half of cron jobs on backend instances to take advantage s of 9 free hours.

如果您不知道如何减少实例时间，请提出建议.

If u have any idea how to reduce the instance hours, please advice.

推荐答案

是的，时间表重叠的cron作业可能会导致不必要的活动高峰，从而增加实例时间.在分钟开始时，不仅每分钟平均有3个请求，而且应用程序的请求队列中同时有3个请求 .如果您使用自动/基本缩放配置，则GAE可能会决定生成其他实例以保持较低的请求延迟，即使单个实例可以轻松地平均每分钟处理3个请求.

Yes, cron jobs with overlapping schedules can cause potentially unnecessary peaks of activity which can drive up the instance hours. Not only that you'd have an average of 3 requests per minute, you'd have 3 requests in your app's request queue simultaneously, at the beginning of that minute. If you use automatic/basic scaling configs GAE may decide to spawn additional instances to keep the request latency low, even if a single instance could easily handle the average of 3 requests per minute.

侧面说明:您已经收到cron请求来触发作业，您有点不必要地创建了另一个(任务队列)请求来实际执行作业(因此，在您的情况下，每分钟6个请求)，为什么不直接从cron处理程序中执行作业呢?

Side note: you already have the cron request coming in to trigger the job, you're kinda unnecessarily creating yet another (task queue) request to actually execute the job (thus amounting to 6 requests per minute in your case), why not just execute the job directly from the cron handler?

现在，如果每个请求需要大约1分钟才能完成交错，将无济于事-@MatrixTai的评论适用.

Now, if each of the requests takes close to ~1 minute to complete staggering them won't help much - @MatrixTai's comment applies.

但是，如果所有3个作业的总执行时间少于1分钟，则错开它们将有助于降低成本.在这种情况下，也许更好的方法是不要再创建3个请求-缺点是您仍然需要仔细配置延迟以确保它们不重叠，而只是顺序执行这3个作业，而不会使新请求排队(这就是我的工作):

But if the combined execution time of all 3 jobs is below 1 minute staggering them will help keep the costs down. Maybe a better approach in this case would be to not create 3 more requests - which has the disadvantage that you still need to carefully configure the delays to ensure they don't overlap, but simply sequentially execute the 3 jobs, without enqueueing new requests (this is what I do):

@app.route('/per_minute_jobs/')
def handle_per_minute_jobs():  # OK if the combined execution time is < 1 min
     execute_job_1()
     execute_job_2()
     execute_job_3()

使用这种方法，您每分钟恰好有1个请求，就其本身而言，它不能像其他方法那样触发生成新实例.

With this approach you have exactly 1 request per minute, which, by itself, can't trigger spawning new instances as the other approaches did.

现在，合并的作业执行可能比典型的(非cron)请求处理花费更长的时间.如果您在同一个GAE服务中同时处理这两个实例，那么即使您的应用程序总体上的平均流量很低，以至于显然仅用一个实例即可处理它们，您仍将至少运行2个实例

Now the combined job execution could take significantly longer than your typical (non-cron) request handling. If you handle them both in the same GAE service, you'll still end up with at least 2 instances running, even if overall your app has low enough average traffic that could apparently be handled with just one instance.

更新:

好吧，由于您必须将工作拆分为后续任务，因此只需使用deferred.defer()的可选countdown或eta args来及时错开这些任务并平滑那些活动高峰.

OK, since you have to split the work in subsequent tasks, just use the deferred.defer()'s optional countdown or eta args to stagger those tasks in time and smooth those activity peaks.

这篇关于合并cron作业以减少实例数量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！