Spark Structure Streaming作业在集群模式下失败 [英] Spark Structure Streaming job failing in cluster mode

问题描述

我在应用程序中使用spark-sql-2.4.1 v.

I am using spark-sql-2.4.1 v in my application.

在将数据写入hdfs文件夹时，我在Spark流应用程序中遇到了这个问题

While writing data on to hdfs folder I am facing this issue in spark-streaming application

错误:

    yarn.Client: Deleted staging directory hdfs://dev/user/xyz/.sparkStaging/application_1575699597805_47
    20/02/24 14:02:15 ERROR yarn.Client: Application diagnostics message: User class threw exception: org.apache.hadoop.security.AccessControlException: Permission denied: user= xyz, access=WRITE, inode="/tmp/hadoop-admin":admin:supergroup:drwxr-xr-x
    .
    .
    .
    Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=xyz, access=WRITE, inode="/tmp/hadoop-admin":admin:supergroup:drwxr-xr-x
            at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:350)
            at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:251)

在将数据写入HDFS文件夹时，当我在yarn-cluster模式下运行时，我在spark-streaming应用程序中遇到此问题时 即

While writing data on to HDFS folder I am facing this issue in spark-streaming application when I run in yarn-cluster mode I face this issue i.e.

    --master yarn \
    --deploy-mode cluster \

但是当我在"yarn-client"模式下运行时，它运行良好 即

But when I run in "yarn-client" mode its run fine i.e.

    --master yarn \
    --deploy-mode client \

任何人都遇到此错误，并且知道此问题的根源是什么?

Anyone come across this error and know what is the root-case of this problem?

这里的基本问题，为什么要尝试用"/tmp/hadoop-admin/"而不是相应的用户目录(即hdfs://qa2/user/xyz/ ??

Fundamental question here , why it is trying to write in "/tmp/hadoop-admin/" instead of respective user directory i.e. hdfs://qa2/user/xyz/ ??

我已经解决了这个问题

https://issues.apache.org/jira/browse/SPARK-26825

但是不确定如何在我的spark-sql应用程序中实现它. 任何帮助都将受到高度赞赏.

But not sure how to implement it in my spark-sql application. Any help is highly appreciated.

推荐答案

有效的--deploy-mode client和失败的--deploy-mode cluster案例之间的唯一区别是驱动程序的位置.在客户端部署模式下，驱动程序在您执行spark-submit的计算机上运行(通常是配置为使用YARN群集的边缘节点，但它不是其中的一部分)，而在群集部署模式下，驱动程序则作为一部分运行YARN群集(受YARN控制的节点之一)的数量.

The only difference between the working --deploy-mode client and the failing --deploy-mode cluster cases is the location of the driver. In client deploy mode, the driver runs on the machine you execute spark-submit (which is usually an edge node that is configured to use a YARN cluster, but it is not part of it) while in cluster deploy mode the driver runs as part of a YARN cluster (one of the nodes under control of YARN).

您似乎配置了错误的边缘节点.

It looks like you've got a misconfigured edge node.

如果常规的仅基于Spark SQL的Spark应用程序也会失败，我也不会感到惊讶.听说它与流查询(Spark Structured Streaming)无关，并且对于任何Spark应用程序都将失败，我不会感到惊讶.

I'd not be surprised if a regular Spark SQL-only Spark application would be failing too. I'd not be surprised to hear that it has nothing to do with a streaming query (Spark Structured Streaming) and would fail for any Spark application.

这篇关于Spark Structure Streaming作业在集群模式下失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！