所有bq作业的摘要 [英] summary of all bq jobs

问题描述

有没有一种方法可以使用bq命令行工具列出给定时间范围内的所有作业ID?我需要做的是遍历所有Id，查找是否有任何错误.

Is there a way to list all job id's using bq command line tool for a given timeframe? What I need to do is to loop through all Id's and find if there is any error.

我使用Web界面知道工作ID，然后使用以下命令:

I use the web interface to know the job id and then use the command:

bq show -j --format=prettyjson job_id

稍后，我将手动复制粘贴输出的错误"部分.这需要花费大量时间来报告给定日期的工作摘要.

Later I would manually copy paste the "error" part of the output. This takes a lot of time to report the job summary for a given day.

推荐答案

当然，您可以通过运行以下命令列出您有权访问的项目的最后1,000个作业:

Sure, you can list up to the last 1,000 jobs for a project you have access to by running:

bq  ls -j --max_results=1000 project_number

如果您有超过1,000个作业，则还可以编写Python脚本，以分批浏览1,000个结果来列出所有作业，如下所示:

If you have more than 1,000 jobs, you can also write a Python script to list all jobs by paging through results in batches of 1,000 - like so:

import httplib2
import pprint
import sys

from apiclient.discovery import build
from apiclient.errors import HttpError

from oauth2client.client import AccessTokenRefreshError
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.client import flow_from_clientsecrets
from oauth2client.file import Storage
from oauth2client.tools import run


# Enter your Google Developer Project number
PROJECT_NUMBER = 'XXXXXXXXXXXX'

FLOW = flow_from_clientsecrets('client_secrets.json',
                               scope='https://www.googleapis.com/auth/bigquery')



def main():

  storage = Storage('bigquery_credentials.dat')
  credentials = storage.get()

  if credentials is None or credentials.invalid:
    credentials = run(FLOW, storage)

  http = httplib2.Http()
  http = credentials.authorize(http)

  bigquery_service = build('bigquery', 'v2', http=http)
  jobs = bigquery_service.jobs()

  page_token=None
  count=0

  while True:
    response = list_jobs_page(jobs, page_token)
    if response['jobs'] is not None:
      for job in response['jobs']:
        count += 1
        print '%d. %s\t%s\t%s' % (count,
                                  job['jobReference']['jobId'],
                                  job['state'],
                                  job['errorResult']['reason'] if job.get('errorResult') else '')
    if response.get('nextPageToken'):
      page_token = response['nextPageToken']
    else:
      break


def list_jobs_page(jobs, page_token=None):
  try:
    jobs_list = jobs.list(projectId=PROJECT_NUMBER,
                          projection='minimal',
                          allUsers=True,
                          maxResults=1000,
                          pageToken=page_token).execute()

    return jobs_list

  except HttpError as err:
    print 'Error:', pprint.pprint(err.content)


if __name__ == '__main__':
  main()

这篇关于所有bq作业的摘要的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！