具有IntelliJ和SBT的自定义文件夹结构的Uber jar [英] Uber jar with custom folder structure with IntelliJ and SBT
问题描述
我对云和SBT/IntelliJ
还是很陌生,因此请尝试在IntelliJ & SBT
构建环境下运气,将我的jar部署在数据proc集群上.
I am fairly new to cloud and SBT/IntelliJ
, So trying my luck with IntelliJ & SBT
build environment to deploy my jar on data proc cluster.
这是我的项目结构的屏幕截图:
Here's a screen shot of my project structure:
代码很简单,它的主要内容定义在'mytestmain'
中,它调用了'ReadYamlConfiguration'
中定义的另一个方法,该方法需要一个moultingyaml
依赖关系,如我在build.sbt
中所示.
Code is quite simple with main defined in 'mytestmain'
which call another method defined in 'ReadYamlConfiguration'
which needed a moultingyaml
dependency, which I have included as shown in my build.sbt
.
这是我的build.sbt
& assembly.sbt
文件:
Here's my build.sbt
& assembly.sbt
file:
lazy val root = (project in file(".")).
settings(
name := "MyTestProjectNew",
version := "0.0.1-SNAPSHOT",
scalaVersion := "2.11.12",
mainClass in Compile := Some("com.test.processing.jobs.mytestmain.scala")
)
libraryDependencies ++= Seq(
"net.jcazevedo" %% "moultingyaml" % "0.4.2"
)
scalaSource in Compile := baseDirectory.value / "src"
assembly.sbt
文件:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10")
我创建了assembly.sbt
来创建Uber jar,以便包括所需的依赖关系,并从Terminal运行了'SBT assembly'
.它已成功创建了一个程序集jar文件,该文件可以在Dataproc群集上成功部署和运行.
I created assembly.sbt
to create Uber jar in order to include required dependencies and ran 'SBT assembly'
from Terminal. It has created a assembly jar file successfully, Which I was able to deploy and run successfully on Dataproc cluster.
gcloud dataproc jobs submit spark \
--cluster my-dataproc-cluster \
--region europe-north1 --class com.test.processing.jobs.mytestmain \
--jars gs://my-test-bucket/spark-jobs/MyTestProjectNew-assembly-0.0.1-SNAPSHOT.jar
代码按预期工作正常,没有问题.
Code is working fine as expected with no issues.
现在,我想拥有自己的自定义目录结构,如下所示:
Now I would like to have my own custom directory structure as shown below:
例如,我想要一个名为'spark-job'
的文件夹,其子目录名为'SparkDataProcessing'
,然后是一个src/main/scala
文件夹,其中包含软件包以及相应的scala类和对象等.
For example, I would like to have a folder name as 'spark-job'
with a sub dir named as 'SparkDataProcessing'
and then src/main/scala
folder with packages and respective scala classes and objects etc.
我的主要方法在'com.test.processing'
软件包的'job'
软件包中定义.
my main method is defined in in package 'job'
within 'com.test.processing'
package.
我需要在build.sbt
中进行哪些更改?我正在根据我的项目结构使用build.sbt
作为示例来寻找详细说明.另外,请提出gitignore
文件中需要包含的所有内容.
What all changes do I need to make in build.sbt
? I am looking for a detail explanation with build.sbt
as a sample according to my project structure. Also please suggest what all needs to be included in gitignore
file.
我正在使用IntelliJ Idea 2020 community edition
和SBT 1.3.3
版本.我在这里到那里尝试了很少的东西,但是总是以结构,jar或build.sbt
问题结束一些问题.
我期待在下面的帖子中找到类似的答案.
I am using IntelliJ Idea 2020 community edition
and SBT 1.3.3
version. I tried few things here and there but always ended up some issue with structure, jar or build.sbt
issues.
I was expecting an answer something similar which is done in below post.
为什么我的sourceDirectories设置对sbt没有影响?
如下面的图片所示,源目录已更改.
As you can see in below pic, the source directory has been changed.
spark-jobs/SparkDataProcessing/src/main/Scala
当我使用下面的路径构建它时,它不起作用.
and when I am building this with below path, it's not working.
scalaSource in Compile := baseDirectory.value / "src"
当我保留默认结构时它起作用.就像src/main/scala
it works when I keep the default structure. like src/main/scala
推荐答案
您还需要在受影响文件顶部的package
关键字之后更改程序包名称.但是,如果您使用IntelliJ重构(通过创建程序包,然后使用UI将文件拖到程序包中),则IntelliJ将为您完成此操作.
You also need to change the package name after the package
keyword at the top of affected files. However, if you refactor using IntelliJ (by creating the packages and then dragging the files into the package using the UI), then IntelliJ will do this for you.
无需更改(build.sbt
和相关文件可以保持不变).
Nothing else needs to be changed (build.sbt
and related files can stay the same).
最后,切记更改class
参数以反映入口点位置的变化;您将通过--class com.test.processing.jobs.job.mytestmain
而不是--class com.test.processing.jobs.mytestmain
.
Finally, remember to change the class
argument to reflect changes in entrypoint locations; you would pass --class com.test.processing.jobs.job.mytestmain
instead of --class com.test.processing.jobs.mytestmain
.
.gitignore:请查看 gitignore文件示例其中包括:
As for .gitignore: please take a look at an example gitignore file which includes:
- 包含目标"的输出目录
- IntelliJ目录,例如".idea"
另一个gitignore示例忽略了由编译器生成的所有.class文件,这是另一种方法.您应该包括所有动态生成的文件,这些文件的更改对其他开发人员而言无关紧要.
Another gitignore example ignores all .class files generated by the compiler, another approach. You should include all files which are generated dynamically, where changes do not matter to other developers.
这篇关于具有IntelliJ和SBT的自定义文件夹结构的Uber jar的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!