将大型CSV加载到Google BigQuery时出错 [英] Error Loading Large CSV into Google BigQuery

问题描述

在将大型CSV加载到bigquery时出现错误。我在网上阅读的地方都看到，对压缩文件有5GB大小的限制，但对CSV没有限制。

加载操作中的BigQuery错误：处理作业'bqjob_r3016bbfad3037f_0000015cea1a1eff_1'时出错：输入CSV文件不可拆分，
至少有一个文件大于最大值允许的大小。大小为：24686861596.最大允许大小为：4294967296。

解决方案

BigQuery文档列出了导入作业的各种限制： https://cloud.google.com/bigquery/quota-policy#import
尤其值得注意的是，压缩CSV文件的限制为4 GB。

有关不可拆分CSV文件的错误消息有两种情况： / p>

1. CSV文件被压缩


2. 其中一个字段中存在引号字符不匹配，它在该字段中看起来像非常长字符串，也使文件不可拆分（这是您的情况可能发生的情况）。

Getting an Error on loading a large CSV into bigquery. Everywhere I read online I see that there is a 5gb size limit on zipped files but no limits on CSV.

BigQuery error in load operation: Error processing job 'bqjob_r3016bbfad3037f_0000015cea1a1eff_1': Input CSV files are not splittable and at least one of the files is larger than the maximum allowed size. Size is: 24686861596. Max allowed size is: 4294967296.

解决方案

BigQuery documentation lists various limits for import jobs here: https://cloud.google.com/bigquery/quota-policy#import In particular it notes, that the limit of compressed CSV file is 4 GBs.

The error message about "not splittable" CSV file can come in two cases:

1. CSV file was compressed
2. There is a quoting character mismatch in one of the fields, which makes it look like very long string in that field, also making file not splittable (this is what likely happened in your case).

这篇关于将大型CSV加载到Google BigQuery时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！