如何在 Zeppelin 中使用来自 S3 的依赖项? [英] How to use dependencies from S3 in Zeppelin?

问题描述

有没有办法添加 S3 上存储桶中的 jar 作为 Zeppelin 的依赖项?尝试了 z.load(s3n://...) 和 z.addRepo(some_name).url(s3n://...) 但他们没有似乎可以胜任..

Is there a way to add jars that are in a bucket on S3 as a dependency of Zeppelin? tried z.load(s3n://...) and z.addRepo(some_name).url(s3n://...) but they don't seem to do the job..

推荐答案

您可以从 S3 下载 jar 并将其放在本地 FS 上.它可以在 %dep 解释器中完成，如下所示:

You could download jars from S3 and put it on the local FS. It could be done inside %dep interpreter like this:

%dep
import com.amazonaws.services.s3.AmazonS3Client
import java.io.File
import java.nio.file.{Files, StandardCopyOption}

val dest = "/tmp/dependency.jar"
val s3 = new AmazonS3Client()
val stream = s3.getObject("buckename", "path.jar").getObjectContent

Files.copy(stream, new File(dest).toPath, StandardCopyOption.REPLACE_EXISTING)

z.load(dest)

注意:您必须生成fat jar，即包含默认情况下未提供的所有自定义依赖项(例如，当您的项目中有多个模块时).在 maven 中可以使用 maven-shade-plugin 来实现:

Note: You must generate fat jar, i.e. include all custom dependencies not provided by default (for example when you have multiple modules in your project). In maven it could be implemented with maven-shade-plugin:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.4.2</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <artifactSet>
                    <includes>
                        <include>com.yourcompany:*</include>
                    </includes>
                </artifactSet>
            </configuration>
        </execution>
    </executions>
</plugin>

这篇关于如何在 Zeppelin 中使用来自 S3 的依赖项?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！