度量系统无法识别应用程序jar中的自定义源/接收器 [英] Metrics System not recognizing Custom Source/Sink in application jar

问题描述

在此处的跟踪.

我在应用程序jar中添加了自定义源"和接收器"，并找到了一种在独立集群节点上获取静态固定的metrics.properties的方法.当我要启动我的应用程序时，我提供了静态路径-spark.metrics.conf ="/fixed-path/to/metrics.properties".尽管我的代码/fat-jar中包含我的自定义源代码/接收器，但在CustomSink上却收到ClassNotFoundException.

我的胖子(带有自定义源代码/接收器代码)在具有对所有人的读取访问权限的hdfs上.

这就是我已经尝试设置的所有内容(因为执行者无法在胖胖的应用程序中找到自定义源/接收器")

1. spark.executor.extraClassPath = hdfs://path/to/fat-jar
2. spark.executor.extraClassPath = fat-jar-name.jar
3. spark.executor.extraClassPath = ./fat-jar-name.jar
4. spark.executor.extraClassPath = ./
5. spark.executor.extraClassPath =/dir/on/cluster/*(尽管*不在文件级别，但目录更多-我无法知道随机的application-id或driver-id来给出绝对名称，启动应用程序)

在这种情况下，执行者似乎是这样初始化的(如果我错了，请纠正我)-

1. 驱动程序告诉这是jar的位置-hdfs://../fat-jar.jar，这是一些属性，例如spark.executor.memory等.
2. N个执行器在群集上旋转(取决于配置)
3. 开始下载hdfs://../fat-jar.jar，但同时会初始化指标系统(?-不确定此步骤)
4. 度量系统正在寻找自定义接收器/源文件-因为它在metrics.properties中已提及-甚至在完成下载fat-jar(实际上具有所有这些文件)之前(这是我的假设)
5. ClassNotFoundException-未找到CustomSink！

我的理解正确吗?此外，还有什么我可以尝试的吗?如果有人有自定义源/接收器的经验，将不胜感激.

解决方案

当我需要扩展现有的 GraphiteSink 类时，我偶然发现了同一个ClassNotFoundException，这是我能够解决的方法.

首先，我在 org.apache.spark.metrics.sink 包中创建了一个 CustomGraphiteSink 类:

  package org.apache.spark.metrics.sink;公共类CustomGraphiteSink扩展了GraphiteSink {} 

然后我在 metrics.properties 中指定了该类 *.sink.graphite.class = org.apache.spark.metrics.sink.CustomGraphiteSink

并通过以下方式将此文件传递给spark-submit:-conf spark.metrics.conf = metrics.properties

Followup from here.

I've added Custom Source and Sink in my application jar and found a way to get a static fixed metrics.properties on Stand-alone cluster nodes. When I want to launch my application, I give the static path - spark.metrics.conf="/fixed-path/to/metrics.properties". Despite my custom source/sink being in my code/fat-jar - I get ClassNotFoundException on CustomSink.

My fat-jar (with Custom Source/Sink code in it) is on hdfs with read access to all.

So here's what all I've already tried setting (since executors can't find Custom Source/Sink in my application fat-jar):

1. spark.executor.extraClassPath = hdfs://path/to/fat-jar
2. spark.executor.extraClassPath = fat-jar-name.jar
3. spark.executor.extraClassPath = ./fat-jar-name.jar
4. spark.executor.extraClassPath = ./
5. spark.executor.extraClassPath = /dir/on/cluster/* (although * is not at file level, there are more directories - I have no way of knowing random application-id or driver-id to give absolute name before launching the app)

It seems like this is how executors are getting initialized for this case (please correct me if I am wrong) -

1. Driver tells here's the jar location - hdfs://../fat-jar.jar and here are some properties like spark.executor.memory etc.
2. N number of Executors spin up (depending on configuration) on cluster
3. Start downloading hdfs://../fat-jar.jar but initialize metrics system in the mean time (? - not sure of this step)
4. Metrics system looking for Custom Sink/Source files - since it's mentioned in metrics.properties - even before it's done downloading fat-jar (which actually has all those files) (this is my hypothesis)
5. ClassNotFoundException - CustomSink not found!

Is my understanding correct? Moreover, is there anything else I can try? If anyone has experience with custom source/sinks, any help would be appreciated.

解决方案

I stumbled upon the same ClassNotFoundException when I needed to extend existing GraphiteSink class and here's how I was able to solve it.

First, I created a CustomGraphiteSink class in org.apache.spark.metrics.sink package:

package org.apache.spark.metrics.sink;
public class CustomGraphiteSink extends GraphiteSink {}

Then I specified the class in metrics.properties *.sink.graphite.class=org.apache.spark.metrics.sink.CustomGraphiteSink

And passed this file to spark-submit via: --conf spark.metrics.conf=metrics.properties

这篇关于度量系统无法识别应用程序jar中的自定义源/接收器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！