数据处理作业失败，找不到类异常 [英] Data proc job is failing with Class not found exception

问题描述

我才刚开始学习GCP和IntelliJ SBT.请回答任何基本问题.

I just started learning GCP and IntelliJ SBT for the first time.Please bear with me for any basic question.

我的项目结构:

这是我的SBT.Build

Here is my SBT.Build

name :: ="MySBTproject"

name := "MySBTproject"

version := "0.1"
scalaVersion := "2.11.12"
val moutlingyaml = "net.jcazevedo" %% "moultingyaml" % "0.4.2"
lazy val root = (project in file("."))
  .settings(
    name := "MySBTproject",
    libraryDependencies += moutlingyaml
  )

然后我从Terminal运行SBT包以创建一个jar，如下所示

Then i ran SBT package from Terminal to create a jar as shown below

C:\Users\xyz\IdeaProjects\MySBTproject>SBT Package 

将此罐子部署到GCP存储桶后，我尝试使用数据proc运行该作业

After deploying this jar on to GCP bucket,i tried running the job using data proc

  gcloud dataproc jobs submit spark \
--cluster my-cluster \
--region europe-north1  \
--jars gs://test-my-bucket-01/spark-jobs/mysbtproject_2.11-0.1.jar \
--class com.test.processing.jobs.mytestmain 

我一旦完成工作，就会遇到错误

I am getting below error once i run the job

Job failed with message [java.lang.ClassNotFoundException: com.test.processing.jobs.mytestmain] 

是因为我的自定义项目目录结构和build.sbt不同步吗?

Is it because of my custom project directory structure and build.sbt are not in sync?

是否需要任何更改，还是需要从项目子目录中创建一个jar，如下所示?

Are any changes required, or do i need to create a jar from project sub-directory as shown below?

C:\Users\xyz\IdeaProjects\MySBTproject\ProcessDataDataProcessingJobs>SBT Package

推荐答案

src目录应位于project.in(directory)指向的目录中.在您的情况下，项目目录是ProcessData，而src在ProcessData/DataProcessingJobs中.因此，我猜测sbt根本看不到您的代码，也没有编译它，也没有打包它.

src directory should be in directory pointed by project.in(directory). In your case project directory is ProcessData, while your src is in ProcessData/DataProcessingJobs. So I'm guessing that sbt doesn't see your code at all, doesn't compile it, and doesn't package it.

您可以通过打开JAR(毕竟这只是一个目录中包含类的ZIP文件！)并调用show sourceDirectories来查看sbt在哪里寻找您的代码来进行检查.

You can check it by opening JAR (after all it's just a ZIP file with classes in directories!) and by calling show sourceDirectories to see where sbt is looking for you code.

这篇关于数据处理作业失败，找不到类异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！