Spark 1.6 DirectFileOutputCommitter [英] Spark 1.6 DirectFileOutputCommitter

问题描述

使用pyspark将文本文件保存到S3时遇到问题。我可以保存到S3，但它首先上传到S3上的_temporary，然后继续复制到预期的位置。这显着增加了工作时间。我试图编译一个DirectFileOutputComter，它应该直接写入想要的S3 url，但我无法让Spark使用这个类。

I am having a problem saving text files to S3 using pyspark. I am able to save to S3, but it first uploads to a _temporary on S3 and then proceeds to copy to the intended location. This increases the jobs run time significantly. I have attempted to compile a DirectFileOutputComitter which should write directly to the intended S3 url, but I cannot get Spark to utilize this class.

示例：

someRDD.saveAsTextFile（s3a：// somebucket / savefolder）

这会创建一个

this creates a

s3a：// somebucket / savefolder / _temporary /

s3a://somebucket/savefolder/_temporary/

$ b $ p

目录，然后写入之后，S3复制操作将文件移回到

directory which is then written to after which a S3 copy operation moves the files back to

s3a：// somebucket / savefolder

s3a://somebucket/savefolder

我的问题是，任何人都有DirectFileOutputCommiter的工作罐，或者任何人有经验这个问题。

My question is does anyone have a working jar of the DirectFileOutputCommiter, or if anyone has experience working around this issue.

相关链接：

Relevant Links:

1. https://issues.apache.org/jira/browse/HADOOP-10400


2. https://gist.github.com/aarondav/c513916e72101bbe14ec


3. https：//mail-archives.apache。 org / mod_mbox / spark-user / 201503.mbox /％3C029201d06334 $ a0871180$e1953480$@gmail.com%3E


4. http://tech.grammarly.com/blog/posts/Petabyte-Scale-Text-Processing -with-Spark.html

1. https://issues.apache.org/jira/browse/HADOOP-10400
2. https://gist.github.com/aarondav/c513916e72101bbe14ec
3. https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3C029201d06334$a0871180$e1953480$@gmail.com%3E
4. http://tech.grammarly.com/blog/posts/Petabyte-Scale-Text-Processing-with-Spark.html

推荐答案

我能解决这个问题通过用Databricks中的DirectOutputCommitter修补Hadoop 2.7.2并将修补过的jar部署到我的spark中nstances。下面链接是一个Git仓库补丁。

I was able to fix this issue by patching Hadoop 2.7.2 with a DirectOutputCommitter from Databricks and deploying the patched jar to my spark instances. Linked below is a git repo with the patched jar.

Github链接

这篇关于Spark 1.6 DirectFileOutputCommitter的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！