Scala Spark-覆盖实木复合地板文件无法删除文件或目录 [英] Scala Spark - overwrite parquet file failed to delete file or dir

问题描述

我试图在本地创建几天的镶木地板文件.第一次运行代码时，一切正常.第二次它无法删除文件.第三次它未能删除另一个文件.无法删除文件是完全随机的.

I'm trying to create parquet files for several days locally. The first time I run the code, everything works fine. The second time it fails to delete a file. The third time it fails to delete another file. It's totally random which file can not be deleted.

我需要执行此操作的原因是因为我想在过去的七天内每天创建镶木地板文件.因此，应该使用更新后的数据覆盖已经存在的镶木地板文件.

The reason I need this to work is because I want to create parquet files everyday for the last seven days. So the parquet files that are already there should be overwritten with the updated data.

我使用Project SDK 1.8，Scala版本2.11.8和Spark版本2.0.2.

I use Project SDK 1.8, Scala version 2.11.8 and Spark version 2.0.2.

第二次运行该行代码后:

After running that line of code the second time:

newDF.repartition(1).write.mode(SaveMode.Overwrite).parquet(
    OutputFilePath + "/day=" + DateOfData)

发生此错误:

WARN FileUtil: 
Failed to delete file or dir [C:\Users\...\day=2018-07-15\._SUCCESS.crc]: 
it still exists.
Exception in thread "main" java.io.IOException: 
Unable to clear output directory file:/C:/Users/.../day=2018-07-15 
prior to writing to it
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:91)

第三次之后:

WARN FileUtil: Failed to delete file or dir 
[C:\Users\day=2018-07-20\part-r-00000-8d1a2bde-c39a-47b2-81bb-decdef8ea2f9.snappy.parquet]: it still exists.
Exception in thread "main" java.io.IOException: Unable to clear output directory file:/C:/Users/day=2018-07-20 prior to writing to it
    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:91)

如您所见，它是第二次运行代码时的另一个文件. 依此类推...手动删除文件后，可以创建所有镶木地板文件.

As you see it's another file than the second time running the code. And so on.. After deleting the files manually all parquet files can be created.

有人知道这个问题以及如何解决吗?

Does somebody know that issue and how to fix it?

编辑:它始终是无法删除的crc文件.

It's always a crc-file that can't be deleted.

推荐答案

感谢您的回答. :) 解决方案是不写在Users目录中.似乎存在权限问题.因此，我在C:目录中创建了一个新文件夹，它可以正常工作.

Thanks for your answers. :) The solution is not to write in the Users directory. There seems to be a permission problem. So I created a new folder in the C: directory and it works perfect.

这篇关于Scala Spark-覆盖实木复合地板文件无法删除文件或目录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！