分布式缓存 [英] distributed cache

问题描述

我在opensuse linux上使用hadoop 19，我没有使用任何集群而是在我的机器上运行我的hadoop代码。我遵循放入分布式缓存的标准技术，但不是一次又一次地从分布式缓存中获取文件，而是将文件的内容存储在一个数组中。从文件中提取的这部分是在configure（）函数中完成的。当我尝试使用fiel名称时，我得到nullPointerException。这是代码的一部分：

i am working with hadoop 19 on opensuse linux, i am not using any cluster rather running my hadoop code on my machine itself. i am following the standard technique on putting in distributed cache but instead of acessing the files from the distributed cache again and again, i stored the contents of the file in an array. this part of extracting from the file is done in the configure() function. i am getting nullPointerException when i try to use the fiel name. this is the part of the code:

.
..part of main()
..
 DistributedCache.addCacheFile(new URI("/home/hmobile/hadoop-0.19.2/output/part-00000"), conf2);
             DistributedCache.addCacheFile(new URI("/home/hmobile/hadoop-0.19.2/output/part-00001"), conf2);
.

.part of mapper

public void configure(JobConf conf2)
{
      String wrd; String line;     try {
                localFiles = DistributedCache.getLocalCacheFiles(conf2);
                System.out.println(localFiles[0].getName());// error NULLPOINTEREXCEPTION
            } catch (IOException ex) {
                Logger.getLogger(blur2.class.getName()).log(Level.SEVERE, null, ex);
            }
            for(Path f:localFiles)// error NULLPOINTEREXCEPTION
            {
                 if(!f.getName().endsWith("crc"))
                 {
                    BufferedReader br = null;
                    try {
                        br = new BufferedReader(new FileReader(f.toString()));

可以在configure（）中完成此类处理吗？

can such processing be not done in configure()?

推荐答案

这将取决于您是否使用本地作业运行器（mapred.job.tracker = local）或者您是否以伪分布式模式运行（即mapred.job.tracker = localhost：8021或= mynode.mydomain.com:8021）。分布式缓存 NOT 在本地模式下工作，只在伪分布式和完全分布式模式下工作。

This will depend on if you're using the local job runner (mapred.job.tracker=local) or if you're running in pseudo-distributed mode (i.e. mapred.job.tracker=localhost:8021 or =mynode.mydomain.com:8021). The distributed cache does NOT work in local mode, only pseudo-distributed and fully distributed modes.

在configure（）中使用分布式缓存是好的，否则。

Using the distributed cache in configure() is fine, otherwise.

这篇关于分布式缓存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！