克朗bigquery工作 [英] Cron bigquery jobs

问题描述

BigQuery没有提供直接的方法，而我从搜索中获得的最好结果是使用应用程序引擎cron服务，但根据我的理解，我必须创建一个Web应用程序才能使用此服务。

我的用例是对点击次数和展示次数进行一些汇总，每天或每周在我们的管理员门户中使用它们。

我之前使用Hive作为数据仓库，而Oozie作为我们的调度程序使用。

有没有一种方法可以用BigQuery来实现相同的逻辑？

解决方案

遗憾的是，尽管工程团队接受请求，但BigQuery中没有内置调度程序！ 链接。

然而，那里是一些有趣的选择。

1. 正如您所提到的，使用App Engine的cron服务绝对有效，您可以编写一个简单的小型Web服务来调用你想要的正常节奏的查询。这项服务不会面向网络，所以收费应该保持在非常低的水平。 Apache Airflow 是我一直在玩的服务，这是非常有前途的;它允许您在Py​​thon中的各种云服务中定义更复杂的数据操作任务，并根据您选择的节奏执行它们。非常便利。


2. 常规Cron - 如果您有可用的服务器，则可以设置一个基本的cron作业，使用'bq'命令行工具执行任何您想要的查询并保存结果到BQ中的表格。

希望有所帮助！我肯定还有其他的选择，只是想给你几个。

Which is the best way to schedule BigQuery jobs?

BigQuery doesn't offer a direct approach, and the best I got from searching is using app engine cron service, but from what I understood I have to create a web application to use this service.

My use case is to do some aggregations over clicks and impressions, daily or weekly and use them in our admin portal.

I used Hive as a data warehouse before and Oozie as our scheduler.

Is there a way to accomplish the same logic with BigQuery?

解决方案

Unfortunately, there is no built in scheduler within BigQuery, although the engineering team takes requests! link.

However, there are a few interesting alternatives.

1. As you mentioned, using the cron service from App Engine would absolutely work, and you could write a small, simple web service that would invoke the query you want on a regular cadence. This service will not be web facing, so the charges should remain extremely small.
2. Apache Airflow is a service that I have been playing around with that is very promising; it allows you to define more complex data manipulation tasks across a variety of cloud services in Python and execute them on whatever cadence you choose. Very handy.
3. Regular Cron - if you have a server available to you, you could just set up a basic cron job that uses the 'bq' command line tool to execute whatever queries you want and save the results to tables in BQ.

Hope that helps! I'm positive there are other options as well, just wanted to give you a few.

这篇关于克朗bigquery工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！