在 GAE 中使用 gcloud-python [英] Using gcloud-python in GAE

问题描述

我有一堆运行一些 python 代码的小 Raspberry Pi，这些代码使用 gcloud-python 数据存储包直接保存到数据存储(跳过 GAE).这很好用.我现在想使用 Google App Engine 通过网络和移动客户端显示数据.在我的 MacBook 上，我通过 pip 使用安装程序和 gcloud 安装了 GAE.我可以编写一个简单的 python 脚本并直接从终端执行它，该终端能够通过 gcloud 从数据存储区写入和读取 - 这也很好用.

I've got a bunch of little Raspberry Pis running some python code which saves directly to the Datastore (skips GAE) using the gcloud-python datastore package. This works great. I now want to present the data via web and mobile clients using Google App Engine. On my MacBook I installed GAE using the installer and gcloud via pip. I can write a simple python script and execute it directly from the terminal which is able to write and read from the datastore via gcloud - that also works just fine.

但是，当我尝试将相同的代码合并到 GAE 中时，它失败了.根据我的研究，我认为这是一个 PATH 问题，但经过几个小时的各种尝试后，我无法解决这个问题.建议将不胜感激.

However, when I try to incorporate that same code into GAE it fails. Based on my research, I expect that it is a PATH issue but after several hours of various attempts I am unable to resolve this. Suggestions would be appreciated.

我相信这篇文章与我的问题类似:Google App Engine, Change哪个python版本

I believe this post is similar to my issue: Google App Engine, Change which python version

以下是一些可能相关的信息:

Here are some pieces of information which may be relevant:

Python 版本

$ which python
/usr/bin/python

根据引用的堆栈溢出问题，我将 GAE 的首选项设置为具有/usr/bin/python 的 Python 路径.我试过了

Per the referenced Stack Overflow issue I set the preferences of GAE to have a Python Path of /usr/bin/python. I have tried it

$PATH

$ echo $PATH
/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/bin:/Users/sheridangray/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin

来自 Python 解释器的 os.path

$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path
<module 'posixpath' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>

gcloud 安装

$ pip install --upgrade gcloud
Requirement already up-to-date: gcloud in /Library/Python/2.7/site-packages
...

Python 2.7 参考

$ sudo find / -name python2.7
Password:
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Dropbox.app/Contents/Resources/include/python2.7
/Applications/Dropbox.app/Contents/Resources/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/lib/python2.7
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/usr/bin/python2.7
/usr/lib/python2.7

相关代码

from gcloud import datastore

class MainHandler(webapp.RequestHandler):
  def get(self):
     dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)

GAE 日志文件

*** Running dev_appserver with the following flags:
--skip_sdk_update_check=yes --port=8080 --admin_port=8000
Python command: /usr/bin/python
INFO     2014-11-21 09:02:03,276 devappserver2.py:745] Skipping SDK update check.
INFO     2014-11-21 09:02:03,288 api_server.py:172] Starting API server at: http://localhost:49183
INFO     2014-11-21 09:02:03,292 dispatcher.py:185] Starting module "default" running at: http://localhost:8080
INFO     2014-11-21 09:02:03,294 admin_server.py:118] Starting admin server at: http://localhost:8000
INFO     2014-11-21 09:02:34,319 module.py:709] default: "GET / HTTP/1.1" 200 2
INFO     2014-11-21 09:02:34,455 module.py:709] default: "GET /favicon.ico HTTP/1.1" 200 8348
INFO     2014-11-21 09:02:44,359 module.py:387] Detected file changes:
  /Users/sheridangray/Projects/city-pulse-web/main.py
ERROR    2014-11-21 09:02:46,443 webapp2.py:1552] gcloud
Traceback (most recent call last):
      File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
  File "/Users/sheridangray/Projects/city-pulse-web/main.py", line 63, in get
dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 103, in get_dataset
connection = get_connection(client_email, private_key_path)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 65, in get_connection
    from gcloud.datastore.connection import Connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/connection.py", line 3, in <module>
    from gcloud import connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 8, in <module>
class Connection(object):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 24, in Connection
USER_AGENT = "gcloud-python/{0}".format(get_distribution('gcloud').version)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 311, in get_distribution
if isinstance(dist,Requirement): dist = get_provider(dist)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 197, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 666, in require
needed = self.resolve(parse_requirements(requirements))
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
DistributionNotFound: gcloud
INFO     2014-11-21 09:02:46,456 module.py:709] default: "GET / HTTP/1.1" 500 5010
INFO     2014-11-21 09:02:46,514 module.py:709] default: "GET /favicon.ico HTTP/1.1" 304 -

推荐答案

您可以在 App Engine 上运行 gcloud-python，但它需要一个一点额外的工作.查看我写的一个骨架项目这让这个工作.

You can run gcloud-python on App Engine, but it requires a little extra work. Check out a skeleton project I wrote which gets this working.

要涵盖的主要基础是:

在install_gcloud.sh

In install_gcloud.sh, pip is used to install gcloud and its dependencies inside the application using

pip install --target="application/vendor/" gcloud

(如其他答案中所述，本地安装不会上传部署到 App Engine.)

(As mentioned in the other answer, local installs don't get uploaded to App Engine on deploy.)

通过pip-安装--target 设置，pkg_resources.get_distribution将按预期工作(它在您的堆栈跟踪中失败).

By pip-installing with --target set, pkg_resources.get_distribution will work as expected (it failed in your stack trace).

另外 pytz 默认情况下有太多读取无法正常工作App Engine，因此使用 gae-pytz 代替.结果，一些pytz导入需要修改.

In addition pytz by default has too many reads to work well on App Engine, so gae-pytz is used instead. As a result, some pytz imports need to be modified.

此外，为了减少 Compute Engine 检查开销(网络开销)，oauth2client.client 模块可以修改.

In addition, to reduce a Compute Engine check overhead (network overhead), the oauth2client.client module can be modified.

这两个调整都可以在一个 提交.

Both of these tweaks can be found in a single commit.

上面的脚本将所有依赖项放在一个名为 vendor/ 的目录中，并且appengine_config.py 将此添加到导入路径中达斯供应商:

The script above puts all dependencies in a directory called vendor/ and appengine_config.py adds this to the import path via Darth Vendor:

import darth
darth.vendor('vendor')

此外，由于 protobuf 依赖也是google 包(就像所有 App Engine 导入一样，例如google.appengine.ext.ndb) 您需要修改 __path__与该包相关联:

In addition, since the protobuf dependency is also part of the google package (just like all the App Engine imports, e.g. google.appengine.ext.ndb) you need to modify the __path__ associated with that package:

import os
import google

curr_dir = os.path.abspath(os.path.dirname(__file__))
vendor_dir = os.path.join(curr_dir, 'vendor')
google.__path__.append(os.path.join(vendor_dir, 'google'))

注意事项(截至 2015 年 1 月 22 日)

请注意，在 App Engine 中使用 gcloud-python比使用原生 App Engine 库 db 或 ndb 慢 3-5 倍.这是因为那些使用直接 RPC 进入 App Engine 运行时，而gcloud-python 将使用 App Engine 之外的 HTTP 与云通信数据存储 API.

CAVEAT (as of January 22, 2015)

Be very aware that using gcloud-python within App Engine will be 3-5x slower than using the native App Engine libraries db or ndb. This is because those use direct RPCs into the App Engine runtime while gcloud-python will use HTTP outside of App Engine to talk to the Cloud Datastore API.

注意:我在最初发布后更新了这个，它引用了以前的历史点.

NOTE: I updated this after the initial posting, which referenced a previous point in history.

这篇关于在 GAE 中使用 gcloud-python的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！