Google Cloud BigQuery Import 在应用引擎项目中不起作用 [英] Google Cloud BigQuery Import not working in app engine project

问题描述

我使用以下代码构建了一个应用引擎项目，以将数据从谷歌云存储桶移动到 bigquery 表中

I have used the following code to build an app engine project to move data from google cloud bucket into the bigquery table

import argparse
import time
import uuid

from google.cloud import bigquery


def load_data_from_gcs(dataset_name, table_name, source):
    bigquery_client = bigquery.Client()
    dataset = bigquery_client.dataset(dataset_name)
    table = dataset.table(table_name)
    job_name = str(uuid.uuid4())

    job = bigquery_client.load_table_from_storage(
        job_name, table, source)

    job.begin()

    wait_for_job(job)

    print('Loaded {} rows into {}:{}.'.format(
        job.output_rows, dataset_name, table_name))


def wait_for_job(job):
    while True:
        job.reload()
        if job.state == 'DONE':
            if job.error_result:
                raise RuntimeError(job.error_result)
            return
        time.sleep(1)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('dataset_name')
    parser.add_argument('table_name')
    parser.add_argument(
        'source', help='The Google Cloud Storage object to load. Must be in '
        'the format gs://bucket_name/object_name')

    args = parser.parse_args()

    load_data_from_gcs(
        args.dataset_name,
        args.table_name,
        args.source)

我还将默认的 app.yaml 文件更改为上述文件并删除了 webapp2 库条目，我的 app.yaml 文件如下所示

I have also altered the default app.yaml file as the above file and deleted the webapp2 library entry and my app.yaml file looks like this

application: gcstobq
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /favicon.ico
  static_files: favicon.ico
  upload: favicon.ico

- url: .*
  script: main.app

由于我是 Python 和应用程序引擎的新手，我不知道是否需要将 main.py 文件中定义的库包含到 app.yaml 中，以及是否需要使用命令行工具运行此应用程序.

As I am new to python and app engine I dont know if I need to include the libraries defines in main.py file into the app.yaml and if i need to run this app using the command line tool.

如果我在这里遗漏了什么，请告诉我?

Please let me know if I am missing something here?

推荐答案

Google Cloud 使用新的 Python 命名空间格式(如果您查看源代码，您会注意到其中没有 __init__.py目录结构).这在 Python 3.3 中用 PEP-420

Google Cloud uses the new Python namespace format (if you look at the source you'll notice that there's no __init__.py in the directory structure). This was changed in Python 3.3 with PEP-420

幸运的是，在 Python 2.7 中，您可以通过避免隐式导入轻松解决此问题.只需将此添加到文件的最顶部(在任何其他导入之前)即可获得 Python 3 行为:

Fortunately in Python 2.7 you can fix this easily by avoiding implicit imports. Just add this to the very top of your file (before any other imports) to get the Python 3 behavior:

from __future__ import absolute_import

这篇关于Google Cloud BigQuery Import 在应用引擎项目中不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！