PySpark 插入覆盖问题 [英] PySpark insert overwrite issue
问题描述
以下是 PySpark ETL 代码的最后两行:
Below are the last 2 lines of the PySpark ETL code:
df_writer = DataFrameWriter(usage_fact)
df_writer.partitionBy("data_date", "data_product").saveAsTable(usageWideFactTable, format=fileFormat,mode=writeMode,path=usageWideFactpath)
其中,WriteMode= append 和 fileFormat=orc
Where, WriteMode= append and fileFormat=orc
我想使用插入覆盖来代替它,以便在我重新运行代码时不会附加我的数据.因此我使用了这个:
I wanted to use insert overwrite in place of this so that my data is not getting appended when I re-run the code. Hence I have used this:
usage_fact.createOrReplaceTempView("usage_fact")
fact = spark.sql("insert overwrite table " + usageWideFactTable + " partition (data_date, data_product) select * from usage_fact")
但这给了我以下错误:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/spark/python/pyspark/sql/session.py", line 545, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/usr/lib/spark/python/pyspark/sql/utils.py", line 69, in deco
raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: u'Cannot overwrite a path that is also being read from.;'
看起来我无法覆盖我正在阅读的路径,但不知道如何纠正它,因为我是 PySpark 的新手.我应该使用什么确切的代码来消除这个问题?
Looks like I cannot overwrite a path from where I am reading from but don't know how to rectify it as I am new to PySpark. What exact code I should use so that this issue is removed?
推荐答案
它对我有用,上面的代码相同.我刚刚在 DDL 中进行了更改,并使用以下详细信息重新创建了表格:(已删除的属性,如果使用)
It worked for me with same above code. I just made change in the DDL and recreated the table with below details: (Removed Properties, if used)
PARTITIONED BY (
`data_date` string,
`data_product` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
WITH SERDEPROPERTIES (
'path'='s3://saasdata/datawarehouse/fact/UsageFact/')
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
LOCATION
's3://saasdata/datawarehouse/fact/UsageFact/'
这篇关于PySpark 插入覆盖问题的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!