Hadoop从Hadoop节点的本地文件系统访问第三方库 [英] Hadoop accessing 3rd party libraries from local file system of a Hadoop node

问题描述

我在 /home/ubuntu/libs/javacv-0.9.jar 的所有Hadoop节点上以及其他一些jar文件中都有一个jar文件。

I have a jar file on all my Hadoop nodes at /home/ubuntu/libs/javacv-0.9.jar , with some other jar files.

当我的 MapReduce 应用程序在 Hadoop 节点上执行时，出现此异常

When my MapReduce application is executing on Hadoop nodes, I am getting this exception

java.io.FileNotFoundException: File does not exist hdfs://192.168.0.18:50000/home/ubuntu/libs/javacv-0.9.jar

如何解决此异常？我在Hadoop中运行的jar如何从Hadoop节点的本地文件系统访问第三方库？

How can I resolve this exception? How can my jar running in Hadoop access 3rd party libraries from the local file system of the Hadoop node?

推荐答案

您需要复制您的文件到HDFS而不是本地文件系统。

You need to copy your file to HDFS and not to the local filesystem.

要将文件复制到HDFS，您需要使用：

To copy files to HDFS you need to use:

hadop fs -put localfile hdfsPath

其他选项是更改文件路径为：

Other option is to change the file path to:

file:///home/ubuntu/libs/javacv-0.9.jar

要将jar文件添加到classpath中，请查看 DistributedCache ：

To add jar files to the classpath, take a look at DistributedCache:

DistributedCache.addFileToClassPath(new Path("file:///home/ubuntu/libs/javacv-0.9.jar"), job);

您可能需要遍历该目录中的所有jar文件。

You may need to iterate over all jar files in that directory.

这篇关于Hadoop从Hadoop节点的本地文件系统访问第三方库的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！