用于存储大量事件的 Google Bigtable 与 BigQuery [英] Google Bigtable vs BigQuery for storing large number of events

问题描述

背景

我们想将不可变事件存储在(最好)托管服务中.一个事件的平均大小小于 1 Kb，我们每秒有 1-5 个事件.存储这些事件的主要原因是，一旦我们创建可能对这些事件感兴趣的未来服务，就能够重放它们(可能使用表扫描).由于我们在 Google Cloud 中，我们显然将 Google 的服务视为首选.

我怀疑

查看诸如

即使无模式数据库更适合我们，我们也可以将我们的事件本质上存储为带有一些元数据的 blob.

问题

我们可以为此使用 BigQuery 而不是 Bigtable 来降低成本吗?例如，BigQuery 有一种叫做 流式插入 的东西在我看来就像我们可以用.如果沿着这条路走下去，我可能不会意识到在短期或长期内有什么会咬我们的东西吗?

解决方案

Bigtable 非常适合大型 (>= 1TB) 可变数据集.它在负载下具有低延迟并由 Google 管理.就您而言，我认为您在使用 BigQuery 时走在正确的轨道上.

Background

We'd like to store our immutable events in a (preferably) managed service. Average size of one event is less than 1 Kb and we have between 1-5 events per second. The main reason for storing these events is to be able to replay them (perhaps using table scanning) once we create future services that might be interested in these events. Since we're in the Google Cloud we're obviously looking at Google's services as first choice.

I suspect that Bigtable would be a good fit for this but according to the price calculator it'll cost us more than 1400 USD per month (which to us is a big deal):

Looking at something like BigQuery renders a price of 3 USD per month (if I'm not missing something essential):

Even though a schema-less database would be better suited for us we would be fine with essentially storing our events as a blob with some metadata.

Questions

Could we use BigQuery for this instead of Bigtable to reduce costs? For example BigQuery has something called streaming inserts which to me seems like something we could use. Is there anything that'll bite us in the short or long term that I might not be aware of if going down this route?

解决方案

Bigtable is great for large (>= 1TB) mutable data sets. It has low latency under load and is managed by Google. In your case, I think you're on the right track with BigQuery.

这篇关于用于存储大量事件的 Google Bigtable 与 BigQuery的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！