在 Dataflow 中自动检测 BigQuery 架构? [英] Autodetect BigQuery schema within Dataflow?

问题描述

是否可以使用等效的 --在 DataFlow 中自动检测?

Is it possible to use the equivalent of --autodetect in DataFlow?

即我们可以在不指定模式的情况下将数据加载到 BQ 表中吗，相当于我们如何使用 --autodetect 从 CSV 加载数据?

i.e. can we load data into a BQ table without specifying a schema, equivalent to how we can load data from a CSV with --autodetect?

(可能相关问题)

推荐答案

如果您使用的是协议缓冲区 作为 PCollections 中的对象(它应该在 Dataflow 后端运行良好)，您可能可以使用我过去编写的 util.它将根据对 protobuffer 描述符的检查，在运行时将 protobuffer 的架构解析为 BigQuery 架构.

If you are using protocol buffers as objects in your PCollections (which should be performing very well on the Dataflow back-end) you might be able to use a util I wrote in the past. It will parse the schema of the protobuffer into a BigQuery schema at runtime, based on inspection of the protobuffer descriptor.

我很快把它上传到了 GitHub，这是 WIP，但你也许能够使用它或受到启发使用 Java 反射编写类似的东西(我可能会在某个时候自己做).

I quickly uploaded it to GitHub, it's WIP, but you might be able to use it or be inspired to write something similar using Java Reflection (I might do it myself at some point).

您可以按如下方式使用该工具:

You can use the util as follows:

TableSchema schema = ProtobufUtils.makeTableSchema(ProtobufClass.getDescriptor());
enhanced_events.apply(BigQueryIO.Write.to(tableToWrite).withSchema(schema)
            .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
            .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));

其中 create 处理将创建具有指定架构的表，ProtobufClass 是使用 Protobuf 架构和 proto 编译器生成的类.

where the create disposition will create the table with the schema specified and the ProtobufClass is the class generated using your Protobuf schema and the proto compiler.

这篇关于在 Dataflow 中自动检测 BigQuery 架构?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！