将BigQuery中的批量数据读取到Datalab中 [英] Reading batches of data from BigQuery into Datalab

问题描述

我在BigQuery表中有一个大数据集(约4500万行，数据为13Gb).我想在Google Datalab Notebook中处理该数据，以便对熊猫进行一些基本统计，以便稍后在Datalab单元中使用matplotlib可视化数据.我认为尝试将所有数据集加载到熊猫的数据帧中不是一个好主意(至少我会遇到RAM问题).

I have a big dataset in BigQuery table (~45M lines, 13Gb of data). I would like to process that data in my Google Datalab Notebook to do some basic statistics with pandas to visualise data later with matplotlib in Datalab cell. I think it is not a good idea to try to load all dataset into pandas' Dataframe (at least I will have RAM issues).

是否可以批量(例如1万行)从BigQuery读取数据以在Datalab中使用它?

Is it possible to read data from BigQuery in batches (say 10K lines) to consume it in Datalab?

提前谢谢！

推荐答案

如果您的目的是可视化数据，抽样会比加载一小批批次更好吗?

If your purpose is to visualize the data, would sampling be better than loading a small batch?

您可以对数据进行采样，例如:

You can sample your data such as:

import google.datalab.bigquery as bq
df = bq.Query(sql='SELECT image_url, label FROM coast.train WHERE rand() < 0.01').execute().result().to_dataframe()

或者，一个使用方便的类:

Or, a use convenient class:

from google.datalab.ml import BigQueryDataSet
sampled_df = BigQueryDataSet(table='myds.mytable').sample(1000)

这篇关于将BigQuery中的批量数据读取到Datalab中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！