在加载文件时Pyspark TextParsingException [英] Pyspark TextParsingException while loading a file

问题描述

我正在使用pyspark加载具有100万条记录的csv文件，但出现错误.TextParsingException:解析输入的长度(1000001)超过了解析器设置中定义的最大字符数(1000000)

I am loading a csv file having 1 million records using pyspark, but getting the error. TextParsingException: Length of parsed input (1000001) exceeds the maximum number of characters defined in your parser settings (1000000)

我检查了文件中的记录是否有大于1000000个字符的数据，但是没有一个记录是这样的.我文件中的最大记录长度是850.

I checked if any of my record in the file has data greater than 1000000 characters, but none of the record is like that. maximum record length in my file is 850.

请帮助....

CODE SNIPPET:
        input_df =  spark.read.format('com.databricks.spark.csv').option("delimiter","\001").option("quote",u"\u0000").load(INPUT_PATH)
        input_df.write.mode('overwrite').format('orc').save(TARGET_LOC)

    SAMPLE DATA

        A    B     C
        --   --    --
        a   xyz"a  123
        b   pqr    456
        c   ABC"z  789

推荐答案

您可以使用

option("maxCharsPerColumn", "-1")

所以现在这应该是这样的:

so this now should work as:

spark.read.format('com.databricks.spark.csv').option("delimiter","\001").option("quote",u"\u0000").option("maxCharsPerColumn", "-1").load(INPUT_PATH)

否则，您也可以尝试更改解析器:

else you can also try changing your parser:

.option("parserLib", "commons")

这篇关于在加载文件时Pyspark TextParsingException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！