我应该如何将blob从BlobStore移到Google云端存储？ [英] How should I move blobs from BlobStore over to Google Cloud Storage?

问题描述

我们的应用程序已经使用Blobstore在App Engine上运行多年。我们希望将我们的视频文件转移到Google云端存储。从Blobstore迁移到GCS的最佳做法是什么？

这只是一个使用BlobReader并一次向GCS写入字节的问题？或者是否有其他快捷方式/工具可用？

至于从App Engine写入GCS，并不缺乏可供选择的库：

• Blobstore API （可以生成BlobKey对于GCS对象，BlobStore Files API已弃用，但一次可使用）


• Google云端存储API （同样不推荐使用，但一次有效）


• 谷歌云存储客户端库


• 的谷歌云存储Python库


• Google云端存储JSON API库


• boto

任何理由使用其中一项？

解决方案

我不必这样做，但我会说没有自动迁移工具。所以是的，你必须推出自己的。

我的方法是使用 cron.yaml 批量迁移，记录哪些文件已迁移，以便您可以为他们服务不同（根据此页面）。在创建对象之后，云存储为您提供了md5散列，您可以使用它来验证每个文件的迁移在删除Blobstore副本之前是否无错误（可以在文件迁移时迁移时计算类似的散列）。

对于图书馆：

• Google云端存储客户端库使用可恢复的上传功能来流文件，这将使内存更平滑。我发现它非常可靠（与已弃用的Google Cloud Storage API / Files API相反）。


• JSON API客户端是较低级别的。
未优化

• boto 在GAE中使用，而在桌面上使用，并且您不希望离开Goog​​le Cloud执行迁移（$$$）。
据我所知，Blobstore允许您从GCS提供文件并让用户将文件上传到GCS，但不会从您的应用程序本身写入文件。 li>

Our application has been running on App Engine using the Blobstore for years. We would like to move our video files over to Google Cloud Storage. What is the best practice for migrating large blobs from Blobstore over to GCS?

Is it just a matter of using BlobReader and writing bytes to GCS one at a time? Or are there other shortcuts/tools available?

As for writing to GCS from App Engine, there are no shortage of libraries to choose from:

• Blobstore API (can generate BlobKey for GCS objects, BlobStore Files API is deprecated, but would be valid for one time use)
• Google Cloud Storage API (again deprecated, but would be valid for one time use)
• Google Cloud Storage Client Library
• Google Cloud Storage Python Library
• Google Cloud Storage JSON API Library
• boto

Any reason to use one over the other?

解决方案

I haven't had to do this, but I'd say there are no automatic migration tools. So yes, you have to roll your own.

My approach would be to batch migrations using cron.yaml, keeping track of which files are migrated so you can serve them differently (as per this page). Cloud storage provides you with a md5 hash after an object is created, which you could use to validate that the migration of each file was error-free before deleting the Blobstore copy (can compute a similar hash while the file is in-transit on migration).

As for libraries:

• the Google Cloud Storage Client Library uses the resumable upload functionality to "stream" the file which will make things smoother memory-wise. I have found it to be quite reliable (as opposed to the deprecated Google Cloud Storage API/Files API).
• the JSON API Client is lower-level.
• boto isn't optimized for use in GAE, but rather on the desktop, and you don't want to be leaving the Google Cloud to do the migration ($$$).
• as far as I know the Blobstore lets you serve files from GCS and have users upload files to GCS but not write files from your application per se.

这篇关于我应该如何将blob从BlobStore移到Google云端存储？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！