使用 Google App Engine 进行高频数据刷新 [英] High frequency data refresh with Google App Engine

问题描述

我正在使用 GAE 为 android 客户端开发一项服务，需要定期刷新应用程序数据，比如每分钟一次.

I'm developing a service using GAE for android clients and need to refresh application data on a regular basis, say once a minute.

就架构而言，应用程序是这样工作的:

In terms of architecture, this is how the application works:

• 用户启动应用并从在 GAE 上运行的服务中检索市场数据
• GAE 服务本身调用外部 Web 服务来检索市场数据、过滤结果并发送以供用户显示
• 市场价格应每分钟更新一次

我知道 GAE 为自动计划任务提供 cron 作业，但据我所知它不适合这种高频任务(甚至不支持)

I know GAE offers cron jobs for automated scheduled tasks, but from what I understood it's not suitable for such high frequency tasks (or not even supported)

我可以用于此用例的最佳实践/工具是什么?

What's the best practice/tools I can use for this use case?

另外，是否建议无论应用程序是否打开都在后台更新数据?还是在用户启动后立即更新?

Also, is it recommended to update data in the background regardless of application being open? Or just update as soon as the user launches?

我还想知道每分钟提取一次数据是否是正确的方法，还是应该改为推送服务?

I would also like to know if pulling data every minute is the right approach, or should the service push instead?

提前致谢.

推荐答案

它不适合这种高频任务(甚至不支持)" - 这不完全正确.

Cron 作业可以低至 1 分钟的间隔运行，请参阅日程格式:

Cron jobs can run at intervals as low as 1 minute, see The schedule format:

以下是时间表示例:

every 12 hours
every 5 minutes from 10:00 to 14:00
every day 00:00
every monday 09:00
2nd,third mon,wed,thu of march 17:00
1st monday of sep,oct,nov 17:00
1 of jan,april,july,oct 00:00

如果您不需要在特定时间运行重复性作业，但是相反只需要定期运行它，使用表单:

If you don't need to run a recurring job at a specific time, but instead only need to run it at regular intervals, use the form:

every N (hours|mins|minutes) ["from" (time) "to" (time)]

因此，您可以使用 1 分钟的间隔:

So for 1 min interval you can use:

every 1 minutes

如果您需要少于 1 分钟的间隔，您可以使用 延迟库 - 任务可以从排队时刻开始延迟，时间值以秒为单位:

If you need lower than 1 minute intervals you can use the deferred library - tasks can be delayed from the enqueueing moment with time values specified in seconds:

deferred.defer(do_something_expensive, "Foobie bletch", 12,_countdown=30, _queue="myqueue")

deferred.defer(do_something_expensive, "Foobie bletch", 12, _countdown=30, _queue="myqueue")

最后一个问题的答案实际上取决于您希望应用程序的行为方式:在客户端应用程序启动时立即为客户端应用程序提供数据，或者让客户端应用程序等到后端收集数据.

The answer to the last questions really depends on how you want your app to behave: have the data immediately available for the client app when the client app starts or have the client app wait until the backend collects the data.

如果您只是将收集的数据转发给客户端，无论哪种方式都可以，没有常见做法"(​​当然，除了不断更新的更高成本所驱动的之外).但是，如果您还计划提供处理历史数据的结果，您可能需要不断更新(或者可能只是在市场开放时间).

If you're just relaying the collected data to the client either way is fine, there is no "common practice" (other than driven by higher costs for constant updates, of course). But if you plan to also offer results from processing historic data you'll probably have to go with constant updates (or maybe just during the market open hours).

更新:

任务队列优于延迟库，延迟功能可以使用可选的 countdown 或 eta 参数到 taskqueue.add():

The Task Queues are preferable to the deferred library, the deferred functionality is available using the optional countdown or eta arguments to taskqueue.add():

• countdown -- 此任务应该运行或租用的时间(以秒为单位).默认为零.如果出现以下情况，请不要指定此参数您指定了一个 eta.

• countdown -- Time in seconds into the future that this task should run or be leased. Defaults to zero. Do not specify this argument if you specified an eta.

eta -- 一个 datetime.datetime，指定任务应该运行的绝对最早时间.如果出现以下情况，则不能指定此参数倒计时参数已指定.这个说法可以是时间zone-aware 或 time zone-naive，或设置为过去的时间.如果参数设置为无，默认值是现在.对于拉取任务，没有worker 可以在 eta 指定的时间之前租用任务论证.

eta -- A datetime.datetime that specifies the absolute earliest time at which the task should run. You cannot specify this argument if the countdown argument is specified. This argument can be time zone-aware or time zone-naive, or set to a time in the past. If the argument is set to None, the default value is now. For pull tasks, no worker can lease the task before the time indicated by the eta argument.

这篇关于使用 Google App Engine 进行高频数据刷新的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！