Google Bigtable vs BigQuery用于存储大量事件 [英] Google Bigtable vs BigQuery for storing large number of events

问题描述

我们希望将我们的不可变事件存储在（最好）托管服务中。 一个事件的平均大小小于1 Kb，我们每秒钟处理1-5个事件。存储这些事件的主要原因是能够在我们创建可能对这些事件感兴趣的未来服务时重播它们（可能使用表扫描）。由于我们在Google Cloud，因此我们显然将Google的服务视为首选。

我怀疑

查看诸如

即使一个无模式数据库会更适合我们，但我们基本上可以存储我们的事件是一些元数据的blob。

问题

我们可以使用用BigQuery来代替Bigtable来降低成本？例如BigQuery有一些名为流式插入，这对我来说似乎是一件我们可以用。有没有什么会在短期或长期内咬我们，我可能不知道如果沿着这条路线走下去？

解决方案

< Bigtable适用于大型（> = 1TB）可变数据集。它的负载延迟低，由Google管理。在你的情况下，我认为你在BigQuery的正确轨道上。

Background

We'd like to store our immutable events in a (preferably) managed service. Average size of one event is less than 1 Kb and we have between 1-5 events per second. The main reason for storing these events is to be able to replay them (perhaps using table scanning) once we create future services that might be interested in these events. Since we're in the Google Cloud we're obviously looking at Google's services as first choice.

I suspect that Bigtable would be a good fit for this but according to the price calculator it'll cost us more than 1400 USD per month (which to us is a big deal):

Looking at something like BigQuery renders a price of 3 USD per month (if I'm not missing something essential):

Even though a schema-less database would be better suited for us we would be fine with essentially storing our events as a blob with some metadata.

Questions

Could we use BigQuery for since instead of Bigtable to reduce costs? For example BigQuery has something called streaming inserts which to me seems like something we could use. Is there anything that'll bite us in the short or long term that I might not be aware of if going down this route?

解决方案

Bigtable is great for large (>= 1TB) mutable data sets. It has low latency under load and is managed by Google. In your case, I think you're on the right track with BigQuery.

这篇关于Google Bigtable vs BigQuery用于存储大量事件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！