BQ中的转义分隔符 [英] Escaping delimiter in BQ

问题描述

我有大量文件，它们之间用|分隔，但是它们具有|以及字段中的值。 |中的数据已用the进行了转义，但我不认为BQ会选择它，是否可以在不必打开每个文件并进行更新的情况下解决此问题？有2-3000个文件，并且都压缩了，所以一一完成根本不可行。

I have a ton of files which are delimited by |, however, they have | as values in the fields as well. the | in the data has been escaped with \ but I don't think BQ is picking it up, is this something I can fix without having to open every single file, and updating? there are 2-3000 files and are all zipped, so doing it one by one is not at all practical.

推荐答案

1. 将每一行读为整行（CSV，带有奇怪的字符定界符）。


2. 在BigQuery中解析-通过REGEX或JavaScript UDF。

我在这里描述了类似的方法：

I describe a similar approach here:

• https： //medium.com/google-cloud/bigquery-lazy-data-loading-ddl-dml-partitions-and-half-a-trillion-wikipedia-pageviews-cd3eacd657b6

• https://medium.com/google-cloud/bigquery-lazy-data-loading-ddl-dml-partitions-and-half-a-trillion-wikipedia-pageviews-cd3eacd657b6

这篇关于BQ中的转义分隔符的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！