从 Google Cloud Storage 加载 csv 文件时出现 BigQuery 错误 [英] BigQuery error when loading csv file from Google Cloud Storage

问题描述

我正在尝试将保存在 GCS 中的 csv 文件的数据加载到 BigQuery 中.csv 文件采用 UTF-8 格式，包含 7 列.我已经在数据方案中指定了这些列(所有字符串和可为空的)，并且我已经检查了看起来很好的 csv 文件的内容.

I'm trying to load the data of a csv file that is saved in GCS into BigQuery. The csv file is in the UTF-8 format and it contains 7 columns. I've specified these columns in the data scheme (all strings and nullable) and I've checked the contents of the csv file which seems fine.

当我尝试加载数据时，出现以下错误:

When I try to load the data I get the following error:

遇到的错误太多.(错误代码:无效)gs://gvk_test_bucket/sku_category.csv:CSV 表引用列位置 1，但从位置:1750384 开始的行仅包含 1列.(错误代码:无效)

Too many errors encountered. (error code: invalid) gs://gvk_test_bucket/sku_category.csv: CSV table references column position 1, but line starting at position:1750384 contains only 1 columns. (error code: invalid)

奇怪的是该文件只包含 680228 行.

The weird thing is that the file only contains 680228 rows.

当我检查 allow jagged lines 选项时，表格正在生成，但只有第一列填充了整个逗号分隔的字符串.

When I check the allow jagged lines options the table is being generated, but only the first column is filled with the entire comma separated string.

有人可以帮我吗?

示例行

119470,Fashion,Fashion Own,男装,男装品牌其他,正装衬衫,长袖衬衫

119470,Fashion,Fashion Own,Menswear,Menswear Brands Other,Formal Shirts,Long Sleeve Shirts

推荐答案

对我来说，这是换行符和回车符存在的问题，请尝试替换特殊字符.我已经使用下面的代码替换了字符，它解决了加载部分.

For me, it was an issue with the presence of new line and carriage return characters, try replacing the special characters. I have replaced the characters using below code and it resolved the loading part.

df= df.applymap(lambda x: x.replace("
"," "))
df= df.applymap(lambda x: x.replace("
"," "))

我使用了 lambda 函数，因为我不知道在我的情况下哪一列是字符串.如果您确定列，则明智地替换其列.

I have used lambda function as I don't know which column is string in my case. If you are sure about columns then replace its column wise.

尝试替换字符，它也适用于您.

Try to replace the characters and it will work for you as well.

这篇关于从 Google Cloud Storage 加载 csv 文件时出现 BigQuery 错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！