在GAE中使用gcloud-python [英] Using gcloud-python in GAE

查看：556
本文介绍了在GAE中使用gcloud-python的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我有一堆小Raspberry Pis运行一些python代码，它使用gcloud-python数据存储包直接保存到Datastore（跳过GAE）。这很好。我现在想通过使用Google App Engine的网络和移动客户端呈现数据。在我的MacBook上，我使用安装程序安装GAE，通过pip安装gcloud。我可以编写一个简单的python脚本，并直接从能够通过gcloud从数据存储中读写的终端执行它 - 这也可以工作。



然而，当我尝试将相同的代码合并到GAE中时，它会失败。根据我的研究，我认为这是一个PATH问题，但经过几个小时的各种尝试后，我无法解决这个问题。建议将不胜感激。



我相信这篇文章与我的问题类似： Google App Engine，更改python版本



以下是一些可能相关的信息：



Python版本
$ b 

  $ which python 
 / usr / bin / python

通过引用的Stack Overflow问题，我设置了首选项的GAE有一个Python路径/ usr / bin / python。我已经尝试过了

$ PATH 

  $ echo $ PATH 
 / Users / sheridangray / Projects / adt-bundle-mac-x86_64-20140702 / sdk / google-cloud-sdk / bin：/ Users / sheridangray / bin：/ usr / local / bin ：/ usr / bin：/ bin：/ usr / sbin：/ sbin：/ usr / local / bin：/ opt / X11 / bin

Python解释器的os.path 

  $ python 
 Python 2.7.6（默认情况下，2014年9月9日，15:04:36）
在darwin 
上输入help，然后输入GCC 4.2.1兼容的Apple LLVM 6.0（clang-600.0.39） ，版权，信用或许可证以获取更多信息。 
>>> import os 
>>> os.path 
< module'posixpath'from'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>

gcloud安装 

  $ pip install --upgrade gcloud 
 Requirement already up-to-date：gcloud in /Library/Python/2.7/site-packages 
 ...

Python 2.7引用 

  $ sudo find / -name python2.7 
密码：
 /Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7 /include/python2.7 
 /Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7 
 /Applications/Dropbox.app/Contents/Resources/include /python2.7 
 /Applications/Dropbox.app/Contents/Resources/lib/python2.7 
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10 .10.sdk / System / Library / Frameworks / Python.framework / Versions / 2.7 / include / python2.7 
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10 .10.sdk /系统/资源库/框架/ Python.framework / Versions / 2.7 / lib / python2.7 
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/python2。 7 
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/lib/python2.7 
 find：/ dev / fd / 3：不是目录
 find：/ dev / fd / 4：不是目录
 /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 
 / System / Library / Frameworks / Python.framework / Versions / 2.7 / include / python2.7 
 /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7 
 / usr / bin / python2.7 
 /usr/lib/python2.7

相关代码 

  from gcloud import datastore 
 
 class MainHandler（webapp.RequestHandler）：
 def get（self）：
 dataset = datastore.get_dataset（dataset_id，email_address，private_key_file）



日志文件 

  ***使用以下标志运行dev_appserver：
 --skip_sdk_update_check = yes --port = 8080 --admin_port = 8000 
 Python命令：/ usr / bin / python 
 INFO 2014-11-21 09：02：03,276 devappserver2.py:745]跳过SDK更新检查。 
 INFO 2014-11-21 09：02：03,288 api_server.py:172]启动API服务器：http：// localhost：49183 
 INFO 2014-11-21 09：02：03,292调度程序。 py：185]启动模块默认运行在：http：// localhost：8080 
 INFO 2014-11-21 09:02:03,294 admin_server.py:118]启动管理服务器：http：// localhost ：8000 
 INFO 2014-11-21 09:02:34,319 module.py:709]默认：GET / HTTP / 1.1200 2 
 INFO 2014-11-21 09:02:34,455 module .py：709] default：GET /favicon.ico HTTP / 1.1200 8348 
 INFO 2014-11-21 09:02:44,359 module.py:387]检测到文件更改：
 / Users /sheridangray/Projects/city-pulse-web/main.py 
 ERROR 2014-11-21 09：02：46,443 webapp2.py:1552] gcloud 
 Traceback（最近一次调用最后一次）：
文件/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine /lib/webapp2-2.5.2/webapp2.py第1535行，在__call__ 
 rv = self.handle_exception（request，response，e）
文件/ Users / sheridangray / Projects / adt-bundle-mac-x86_64-20140702 / sdk / google-cloud- sdk / platform / GoogleAppEngineLauncher.app / Contents / Resources / GoogleAppEngine-default.bundle / Contents / Resources / google_appengine / lib / webapp2-2.5.2 / webapp2.py，第1529行，在__call__ 
 rv = self。 router.dispatch（请求，响应）
文件/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine -default.bundle / Contents / Resources / google_appengine / lib / webapp2-2.5.2 / webapp2.py，第1278行，在default_dispatcher 
返回route.handler_adapter（请求，响应）
文件/ Users /sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5 .2 / webapp2.py，行1102，在__cal l__ 
 return handler.dispatch（）
文件/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/ Resources / GoogleAppEngine-default.bundle / Contents / Resources / google_appengine / lib / webapp2-2.5.2 / webapp2.py，第572行，发送
返回self.handle_exception（e，self.app.debug）
档案/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/ google_appengine / lib / webapp2-2.5.2 / webapp2.py，第570行，发送
返回方法（* args，** kwargs）
文件/ Users / sheridangray / Projects / city-pulse -web / main.py，第63行，获取
 dataset = datastore.get_dataset（dataset_id，email_address，private_key_file）
文件/System/Library/Frameworks/Python.framework/Versions/2.7/ lib / python2.7 / gcloud / datastore / __ init__.py，第103行，在get_dataset 
 connection = get_connectio n（client_email，private_key_path）
在get_connection $ b $中的第65行文件/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py b from gcloud.datastore.connection import Connection 
文件/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/connection.py，第3行，放在< ;模块> 
来自gcloud导入连接
文件/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py，第8行，位于< module> 
类连接（对象）：
文件/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py，第24行，在Connection 
 USER_AGENT =gcloud-python / {0}。format（get_distribution（'gcloud'）。version）
文件/ Users / sheridangray / Projects / adt-bundle-mac-x86_64-20140702 / sdk /google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py，第311行，位于get_distribution 
如果isinstance（dist，Requirement）：dist = get_provider（dist）
文件/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app /Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py，第197行，在get_provider 
 return working_set.find（moduleOrReq）或require（str（ moduleOrReq））[0] 
文件/ Users / sheridangray / Projects / ad T-束-MAC-x86_64-20140702 / SDK /谷歌云SDK /平台/ GoogleAppEngineLauncher.app /内容/资源/ GoogleAppEngine-default.bundle /内容/资源/ google_appengine / lib目录/ setuptools的-0.6c11 / pkg_resources.py ，第666行，需要
 required = self.resolve（parse_requirements（requirements））
文件/ Users / sheridangray / Projects / adt-bundle-mac-x86_64-20140702 / sdk / google-cloud -sdk / platform / GoogleAppEngineLauncher.app / Contents / Resources / GoogleAppEngine-default.bundle / Contents / Resources / google_appengine / lib / setuptools-0.6c11 / pkg_resources.py，第565行解析
 raise DistributionNotFound（req ）＃XXX在这里放置更多信息
 DistributionNotFound：gcloud 
 INFO 2014-11-21 09:02:46,456 module.py:709]默认：GET / HTTP / 1.1500 5010 
 INFO 2014-11-21 09：02：46,514 module.py:709] default：GET /favicon.ico HTTP / 1.1304  -

$您可以在App Engine上运行 gcloud-python ，但它需要一个名为
一点额外的工作。查看一个框架项目我写了
，这样可以正常工作。

要覆盖的主要依据包括：

获取依赖关系



在 install_gcloud.sh ， pip 用于安装 gcloud 及其$ 

  pip install --target =application / vendor /gcloud 
  / code>

（正如其他本地安装在部署时不会上传
到App Engine。）



修改依赖关系



pip -installing with - target set， pkg_resources .get_distribution
将按预期工作（它在堆栈跟踪中失败）。



另外 pytz 默认情况下，读取过多可在
App Engine上正常运行，因此 <$ c代替使用$ c> gae-pytz 。因此，需要修改一些 pytz
进口。

另外，要减少一个计算引擎检查开销（网络开销），
oauth2client.client 模块可以修改。



这两个调整都可以在一个提交中找到。

使进口工作



上面的脚本将所有依赖关系放在一个名为 vendor / 和
appengine_config.py 通过
将这添加到导入路径 Darth Vendor ：

 导入darth 
 darth.vendor （'vendor'）

另外，由于 protobuf 依赖项也是
<$ c的一部分$ c> google 包（就像所有的App Engine导入一样，例如
google.appengine.ext.ndb ）您需要修改 __路径__
与该包： 

  import os 
 import google 
 
 curr_dir = os.path.abspath （os.path.dirname（__ file__））
 vendor_dir = os.path.join（curr_dir，'vendor'）
 google .__ path __。append（os.path.join（vendor_dir，'google'）） ）



CAVEAT（截至2015年1月22日）



请注意，在App Engine中使用 gcloud-python 将比使用本机App Engine库低$ 3-5x db 或 ndb 。
这是因为那些使用直接RPC进入App Engine运行时，而
gcloud-python 将使用App Engine之外的HTTP与Cloud
Datastore API。





注意：我在首次发布后更新引用
a历史上的点。 p>

I've got a bunch of little Raspberry Pis running some python code which saves directly to the Datastore (skips GAE) using the gcloud-python datastore package. This works great. I now want to present the data via web and mobile clients using Google App Engine. On my MacBook I installed GAE using the installer and gcloud via pip. I can write a simple python script and execute it directly from the terminal which is able to write and read from the datastore via gcloud - that also works just fine.

However, when I try to incorporate that same code into GAE it fails. Based on my research, I expect that it is a PATH issue but after several hours of various attempts I am unable to resolve this. Suggestions would be appreciated.

I believe this post is similar to my issue: Google App Engine, Change which python version

Here are some pieces of information which may be relevant:

Python version

$ which python
/usr/bin/python

Per the referenced Stack Overflow issue I set the preferences of GAE to have a Python Path of /usr/bin/python. I have tried it

$PATH

$ echo $PATH
/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/bin:/Users/sheridangray/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin

os.path from Python interpreter

$ python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path
<module 'posixpath' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc'>

gcloud installation

$ pip install --upgrade gcloud
Requirement already up-to-date: gcloud in /Library/Python/2.7/site-packages
...

Python 2.7 references

$ sudo find / -name python2.7
Password:
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Dropbox.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Dropbox.app/Contents/Resources/include/python2.7
/Applications/Dropbox.app/Contents/Resources/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/python2.7
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/lib/python2.7
find: /dev/fd/3: Not a directory
find: /dev/fd/4: Not a directory
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/usr/bin/python2.7
/usr/lib/python2.7

Relevant Code

from gcloud import datastore

class MainHandler(webapp.RequestHandler):
  def get(self):
     dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)

GAE Log File

*** Running dev_appserver with the following flags:
--skip_sdk_update_check=yes --port=8080 --admin_port=8000
Python command: /usr/bin/python
INFO     2014-11-21 09:02:03,276 devappserver2.py:745] Skipping SDK update check.
INFO     2014-11-21 09:02:03,288 api_server.py:172] Starting API server at: http://localhost:49183
INFO     2014-11-21 09:02:03,292 dispatcher.py:185] Starting module "default" running at: http://localhost:8080
INFO     2014-11-21 09:02:03,294 admin_server.py:118] Starting admin server at: http://localhost:8000
INFO     2014-11-21 09:02:34,319 module.py:709] default: "GET / HTTP/1.1" 200 2
INFO     2014-11-21 09:02:34,455 module.py:709] default: "GET /favicon.ico HTTP/1.1" 200 8348
INFO     2014-11-21 09:02:44,359 module.py:387] Detected file changes:
  /Users/sheridangray/Projects/city-pulse-web/main.py
ERROR    2014-11-21 09:02:46,443 webapp2.py:1552] gcloud
Traceback (most recent call last):
      File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
  File "/Users/sheridangray/Projects/city-pulse-web/main.py", line 63, in get
dataset = datastore.get_dataset(dataset_id, email_address, private_key_file)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 103, in get_dataset
connection = get_connection(client_email, private_key_path)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/__init__.py", line 65, in get_connection
    from gcloud.datastore.connection import Connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/datastore/connection.py", line 3, in <module>
    from gcloud import connection
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 8, in <module>
class Connection(object):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gcloud/connection.py", line 24, in Connection
USER_AGENT = "gcloud-python/{0}".format(get_distribution('gcloud').version)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 311, in get_distribution
if isinstance(dist,Requirement): dist = get_provider(dist)
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 197, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 666, in require
needed = self.resolve(parse_requirements(requirements))
  File "/Users/sheridangray/Projects/adt-bundle-mac-x86_64-20140702/sdk/google-cloud-sdk/platform/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 565, in resolve
    raise DistributionNotFound(req)  # XXX put more info here
DistributionNotFound: gcloud
INFO     2014-11-21 09:02:46,456 module.py:709] default: "GET / HTTP/1.1" 500 5010
INFO     2014-11-21 09:02:46,514 module.py:709] default: "GET /favicon.ico HTTP/1.1" 304 -

解决方案

You can run gcloud-python on App Engine, but it requires a little extra work. Check out a skeleton project I wrote which gets this working.

The main bases to cover are:

Getting the dependencies

In install_gcloud.sh, pip is used to install gcloud and its dependencies inside the application using

pip install --target="application/vendor/" gcloud

(As mentioned in the other answer, local installs don't get uploaded to App Engine on deploy.)

Modifying the dependencies

By pip-installing with --target set, pkg_resources.get_distribution will work as expected (it failed in your stack trace).

In addition pytz by default has too many reads to work well on App Engine, so gae-pytz is used instead. As a result, some pytz imports need to be modified.

In addition, to reduce a Compute Engine check overhead (network overhead), the oauth2client.client module can be modified.

Both of these tweaks can be found in a single commit.

Making Imports Work

The script above puts all dependencies in a directory called vendor/ and appengine_config.py adds this to the import path via Darth Vendor:

import darth
darth.vendor('vendor')

In addition, since the protobuf dependency is also part of the google package (just like all the App Engine imports, e.g. google.appengine.ext.ndb) you need to modify the __path__ associated with that package:

import os
import google

curr_dir = os.path.abspath(os.path.dirname(__file__))
vendor_dir = os.path.join(curr_dir, 'vendor')
google.__path__.append(os.path.join(vendor_dir, 'google'))

CAVEAT (as of January 22, 2015)

Be very aware that using gcloud-python within App Engine will be 3-5x slower than using the native App Engine libraries db or ndb. This is because those use direct RPCs into the App Engine runtime while gcloud-python will use HTTP outside of App Engine to talk to the Cloud Datastore API.

NOTE: I updated this after the initial posting, which referenced a previous point in history.

这篇关于在GAE中使用gcloud-python的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆