如何包装火花斯卡拉应用 [英] how to package spark scala application
本文介绍了如何包装火花斯卡拉应用的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我已经开发出使用SparkSQL和SparkStreaming一个独立的火花Scala的应用程序。
I have developed a standalone spark scala application which uses SparkSQL and SparkStreaming.
这工作正常在Eclipse被配置为火花。
This works fine in Eclipse which is configured for spark.
我在Maven的新手。
I am newbie in maven.
要打包使用maven这个应用程序，我也跟着下面的教程
To package this application using maven, I have followed the below tutorial
的http://ryancompton.net/2014/05/19/sample-pomxml-to-build-scala--jar-with-dependenciesjar/
但最终与下面的错误。
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source (scala-compile-first) on project TweetAnalyzer: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30: Failed to read artifact descriptor for org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source (scala-compile-first) on project TweetAnalyzer: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:347)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:154)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:582)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.1.5:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:104)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: org.apache.maven.plugin.PluginResolutionException: Plugin net.alchim31.maven:scala-maven-plugin:3.1.5 or one of its dependencies could not be resolved: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:204)
at org.apache.maven.plugin.internal.DefaultMavenPluginManager.createPluginRealm(DefaultMavenPluginManager.java:359)
at org.apache.maven.plugin.internal.DefaultMavenPluginManager.setupPluginRealm(DefaultMavenPluginManager.java:327)
at org.apache.maven.plugin.DefaultBuildPluginManager.getPluginRealm(DefaultBuildPluginManager.java:229)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:100)
... 20 more
Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at net.alchim31.maven:scala-maven-plugin:jar:3.1.5 -> org.apache.maven.doxia:doxia-sink-api:jar:1.1.2 -> org.apache.maven.doxia:doxia-logging-api:jar:1.1.2 -> org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:292)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:317)
at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve(DefaultPluginDependenciesResolver.java:192)
... 24 more
Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.codehaus.plexus:plexus-container-default:jar:1.0-alpha-30
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:384)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:231)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:461)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:573)
at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:261)
... 26 more
Caused by: org.apache.maven.model.resolution.UnresolvableModelException: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:170)
at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:219)
at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:898)
at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:750)
at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:309)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:375)
... 31 more
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:459)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:239)
at org.apache.maven.repository.internal.DefaultModelResolver.resolveModel(DefaultModelResolver.java:166)
... 36 more
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.codehaus.plexus:plexus-containers:pom:1.0-alpha-30 from/to central (https://repo.maven.apache.org/maven2): Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$6.wrap(WagonRepositoryConnector.java:1016)
at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$6.wrap(WagonRepositoryConnector.java:1004)
at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:725)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.maven.wagon.TransferFailedException: Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:935)
at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:116)
at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88)
at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61)
at org.eclipse.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:660)
... 4 more
Caused by: org.apache.maven.wagon.providers.http.httpclient.conn.HttpHostConnectException: Connect to 172.20.181.136:8080 [/172.20.181.136] failed: Connection refused
at org.apache.maven.wagon.providers.http.httpclient.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:138)
at org.apache.maven.wagon.providers.http.httpclient.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:365)
at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec.execute(RetryExec.java:85)
at org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.maven.wagon.providers.http.httpclient.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.maven.wagon.providers.http.httpclient.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.execute(AbstractHttpClientWagon.java:756)
at org.apache.maven.wagon.providers.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:854)
... 8 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.maven.wagon.providers.http.httpclient.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at org.apache.maven.wagon.providers.http.httpclient.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)
... 18 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
请提供有关如何包装我的应用程序，并在集群中运行的解决方案。
Please provide a solution on how to package my application and run in cluster.
推荐答案
我用SBT作为我的包和编译工具。随着SBT，我用集结号插件包中的最后一个jar文件的依赖关系。我推荐它。
I use sbt as my package and compiling tool. With sbt, I use "assembly" plugin to package dependencies in the final jar file. I recommend it.
如果你是Maven，我相信树荫插件做类似的工作。见遮阳的使用页面，你会有所启发。 http://maven.apache.org/plugins/maven-shade-plugin /usage.html
If you maven, I believe "shade" plugin do the similar job. See the usage page of shade and you will be inspired. http://maven.apache.org/plugins/maven-shade-plugin/usage.html
基本上，你需要在你的pom.xml文件，因为这改变了建设的一部分：
Basically, you need to change the "build" part in your pom.xml file as this:
<project>
...
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<configuration>
<!-- put your configurations here -->
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
...
</project>
这篇关于如何包装火花斯卡拉应用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
