如何解开由Google AppEngine批量下载器编写的sqlite3数据库 [英] How to unpack sqlite3 database written by Google AppEngine bulk downloader

问题描述

我需要在我的应用的数据存储中下载相当大的（多GB）实体的所有实例。我有足够的磁盘空间来存储实体的数据，但不足以存储批量下载器作为SQLite数据库检索的原始数据和下载器在应用完数据后处理的数据的处理版本我的bulkloader.yaml文件中指定的转换。鉴于此，我相当肯定批量下载操作可以成功检索SQLite数据库，然后在尝试应用转换时失败。

这可能没问题，因为还有另外一个系统可以让我可以移动SQLite数据库，并且我可以将其解压缩。 （我可以使用的另一个系统安装了Python，但没有支持AppEngine工具的版本 - 而且我没有在该机器上升级Python的权限 - 所以我不能直接在那里进行批量下载。）我可以检索我需要的数据如果我可以编写一些Python代码来加载SQLite数据库并读取其结果表，但我无法弄清楚SQLite数据的内容 - 当我使用SQLite模块时连接到数据库并解压缩表的行，除了我感兴趣的数据（我的AppEngine应用程序实际放置在数据存储区中的数据）外，它们似乎还包含元数据。

我知道appcfg.py批量下载过程可以读取这些数据，因为它可以按照我在bulkloader.yaml中指定的方式转换数据，但是我没有找到可以执行此操作的AppEngine工具箱代码这个开箱。任何帮助或指针，将不胜感激。

解决方案

实体存储在下载的SQLite数据库作为编码的协议缓冲区'存储在生产环境和其他地方 - 一个实体 是一个编码的PB，简而言之）。你可以通过使用SDK代码解码实体（ db.proto_to_entity（）等）来自己读出它们，但是将所有东西都设置好会有点麻烦。

相关代码是 bulkloader.py中的ResultDatabase类 - 您可以重复使用以及批量加载程序的其他部分，使您的工作更轻松。

I need to download all instances of fairly large (multi-GB) entity in my app's datastore. I have enough disk space to store the entity's data, but not enough to store both the original data that the bulk downloader retrieves as an SQLite database and the processed version of the data that the downloader writes after applying the transforms specified in my bulkloader.yaml file. Given this, I'm fairly certain that the bulk download operation would successfully retrieve the SQLite database, and then fail when trying to apply the transforms.

This might be okay since there's another system available to which I could move the SQLite database and where I could unpack it. (The other system that's available to me has Python installed but not a version that supports the AppEngine tools -- and I don't have permission to upgrade Python on that machine -- so I cannot do the bulk download directly there.) I could retrieve the data I need if I could write some Python code to load the SQLite database and read its result table, but I cannot figure out what to make of the SQLite data -- when I use the SQLite module to connect to the database and unpack rows of the table, they appear to contain metadata in addition to the data that I'm interested in (the data that my AppEngine app actually placed in the datastore).

I know that the appcfg.py bulk download process can read this data, since it can transform the data in the ways I specify in bulkloader.yaml, but I haven't located the AppEngine toolkit code that does this unpacking. Any help or pointers would be appreciated.

解决方案

Entities are stored in the downloaded SQLite database as encoded Protocol Buffers (the same as they're stored in the production environment, and everywhere else - an entity is an encoded PB, in short). You can read them out yourself by using the SDK code for decoding entities (db.proto_to_entity() etc), but it'll be a bit of work to set everything up.

The relevant code is the ResultDatabase class in bulkloader.py - which you can probably reuse, along with other parts of the bulkloader, to make your job easier.

这篇关于如何解开由Google AppEngine批量下载器编写的sqlite3数据库的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！