将 Google Cloud Storage 数据加载到 bigtable [英] Load Google Cloud Storage data into bigtable
问题描述
是否有一种简单的方法或示例可以将 Google Cloud Storage 数据加载到 bigtable 中?
Is there an easy way or example to load Google Cloud Storage data into bigtable?
我有很多由 pyspark 生成的 json 文件，我希望将数据加载到 bigtable.
I have lots of json files generated by pyspark and i wish to load data into bigtable.
但我找不到一种简单的方法来做到这一点！
But I can not find an easy way to do that!
我已经尝试了 google 的 python 代码-cloud-python 并且它工作正常，但它只是将数据逐行读取到 bigtable 中，这对我来说很奇怪.
I have tried the python code from google-cloud-python and it work fined, but it just read data line by line into bigtable which was strange for me.
任何帮助将不胜感激.
推荐答案
在 Cloud Bigtable 中没有读取数据的简单工具.以下是一些选项:
There is no simple tool to read data in Cloud Bigtable. Here are some options:
- 使用 Dataflow 导入文件.这需要 Java 开发，并学习 Dataflow 编程模型.
- 使用 Python(可能使用 Pyspark)读取这些 json 文件，并使用名为 mutate_rows 批量写入 Bigtable.
- Import the files using Dataflow. This requires java development, and learning the Dataflow programming model.
- Use Python (possibly with Pyspark) to read those json files, and write to Cloud Bigtable using a method called mutate_rows which write to Bigtable in bulk.
仅供参考，我在 Cloud Bigtable 团队工作.我是一名 Java 开发人员，所以我选择 #1.我们的团队一直致力于改善我们的 Python 体验.扩展团队最近添加了一些可靠性改进，以确保 mutate_rows 对大型作业具有弹性.我们还没有任何与 PySpark 或 Apache Beam 的 python SDK 集成的好例子，但他们在我们的雷达范围内.
FYI, I work on the Cloud Bigtable team. I'm a Java developer, so I opt for #1. Our team has been working to improve our python experience. The extended team recently added some reliability improvements to make sure that mutate_rows is resilient for large jobs. We do not yet have any good examples of integrating with PySpark or Apache Beam's python SDK, but they are on our radar.
这篇关于将 Google Cloud Storage 数据加载到 bigtable的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
