如何在 AWS S3 中保存和使用 Spark History Server 日志 [英] How to save and use Spark History Server logs in AWS S3

问题描述

我想在 AWS S3 中记录和查看 Spark History Server 的事件日志.

I want to record and view Event Log of Spark History Server in AWS S3.

以下是spark-defaults.conf中记录的属性.

The following are the properties recorded in spark-defaults.conf.

spark.hadoop.fs.s3a.impl          org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.endpoint    {{endpoint}}
spark.hadoop.fs.s3a.access.key  {{accessKey}}
spark.hadoop.fs.s3a.secret.key  {{secretKey}}
spark.hadoop.fs.s3a.fast.upload true
spark.hadoop.fs.s3a.block.size  268435456
spark.eventLog.enabled            true
spark.eventLog.dir                s3a://{{bucketName}}/{{path}}
spark.history.fs.logDirectory     s3a://{{bucketName}}/{{path}}

但是，在启动spark历史服务器时，出现如下错误.

However, when starting the spark history server, the following error occurs.

20/10/07 14:07:14 INFO S3AFileSystem: Error Message: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: {{requestId}}, AWS Error Code: null, AWS Error Message: Forbidden
20/10/07 14:07:14 INFO S3AFileSystem: HTTP Status Code: 403
20/10/07 14:07:14 INFO S3AFileSystem: AWS Error Code: null
20/10/07 14:07:14 INFO S3AFileSystem: Error Type: Client
20/10/07 14:07:14 INFO S3AFileSystem: Request ID: {{requestId}}
20/10/07 14:07:14 INFO S3AFileSystem: Class Name: com.amazonaws.services.s3.model.AmazonS3Exception
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:296)
        at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: {{requestId}}, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: {{requestId}}
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
        at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976)
        at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:956)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:688)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:71)
        at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:257)
        at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:211)
        at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:207)
        at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:86)
        ... 6 more

使用上述认证信息调用API时，上传和下载效果良好.

Uploading and downloading are performed well when calling APIs using the above authentication information.

我在配置 Spark History Server 时是否遗漏了什么?

Did I miss anything when configuring Spark History Server?

推荐答案

我能够让它工作.基本上你需要将相关的 jars 添加到 Spark_Home/jars 目录中.请找到我对类似问题的详细回答:https://stackoverflow.com/a/65086818/6239561

I was able to get this working. Basically you need to add relevant jars to the Spark_Home/jars directory. Please find my detailed answer for a similar question : https://stackoverflow.com/a/65086818/6239561

这篇关于如何在 AWS S3 中保存和使用 Spark History Server 日志的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！