将数据直接加载到 Google BigQuery 与先通过 Cloud Storage 有什么优缺点? [英] What are the pros and cons of loading data directly into Google BigQuery vs going through Cloud Storage first?

问题描述

Also, is there anything wrong with doing transforms/joins directly within BigQuery? I'd like to minimize the number of components and steps involved for a data warehouse I'm setting up (simple transaction and inventory data for a chain of retail stores.)

解决方案

Loading data via Cloud Storage is the fastest (and the cheapest) way. Loading directly can be done via app (using streaming insert which add some additional cost)

For the doing transformation - if what are you plan/need to do can be done in BigQuery - you should do it in BigQuery :) - it is the best and fastest way of doing ETL. But you should take in account cost of running query (if you not paying Google for slots - it could be 5$ per 1TB scans)

Another good options for complex ETL is using Data Flow - but it can became expensive very quick - in exchange of more flexibility.

这篇关于将数据直接加载到 Google BigQuery 与先通过 Cloud Storage 有什么优缺点?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！