如何将事件流式传输到BigQuery？ [英] How to stream events to BigQuery?
问题描述
我想将事件添加到BigQuery中，以便使用诸如 ModeAnalytics 之类的服务来查看它们。
我不确定要理解BigQuery的完整概念，也许我对此做了错误的假设，但是我想用它来做（类） ）表事件，并根据事件名称添加事件。
这可能类似于新帐户，用户搜索等。 p>
但是我能找到的所有文档都是关于从其他表，csv，json文件等加载大量数据的。
我无法找到一个关于像POST请求那样的数据添加到event表的文档。
我怎么能你应该考虑使用 rel =nofollow>云端pubsub 。我会创建一个非常简单的云端，它将以流模式托管并写入将流媒体插入到BQ中，以便由GAE引发每个PubSub事件。所以基本上GAE会引发事件，数据流将订阅并从PubSub中读取并写入BQ。
正如在下面的讨论中提到的：您还可以在任务队列中写入事件日志并将他们从处理程序中取出，然后直接将它们传送到BQ。我通常避免使用大量生产，因为根据我的经验，此技术通常会创建持久的额外实例（因此额外的实例小时数），尤其是在您要编写多个事件的情况下，但是它是一种简单的方法并避免数据流和pubsub使用，并将所有内容保存到appengine中。
I'd like to add events to BigQuery in order to view them with graph using services like ModeAnalytics.
I'm not sure to grasp the full concept of BigQuery and maybe I'm making wrong assumptions about it, but what I'd like to use it for is have a (kind of) table "events" and add events to it based on the event name.
This could be like "new account", "user search", etc.
But all the documentation I could find was about loading a lot of data from other tables, a csv, a json file, etc.
I was not able to find a documentation about just doing something like a POST request with data to add to the "event" table.
How can I do ?
You should consider using cloud pubsub . I would create a very simple cloud data flow component which will be hosted in streaming mode and will write "streaming inserts" into BQ for each PubSub event raised by GAE. So basically GAE will raise events, dataflow will subscribe and read from PubSub and write to BQ.
As mentioned in the discussions below: you could also write event logs in task queue and dequeue them in a handler and then stream them straight to BQ. I generally avoid this for heavy production use as in my experience this technique generally creates "long lasting" extra instances (hence extra instance hours),especially if you have many events to write, however it is an easy way and avoids dataflow and pubsub usage and keeps every thing contained to appengine.
这篇关于如何将事件流式传输到BigQuery？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
