使用storm在远程集群中运行拓扑时文件创建问题 [英] issue in file creation while running topology in remote cluster using storm

问题描述

我创建了一个拓扑，它应该从一个文件中读取并将其写入一个新文件.我的程序在本地集群中正常运行，但在远程集群中提交时，我没有收到任何错误，但没有创建文件.以下是我在远程集群中提交拓扑的代码:-

I have created a topology which should read from a file and write it to a new file. My program is running properly in local cluster but while submitting in remote cluster i am not getting any error but file is not getting created. below is my code to submit topolgy in remote cluster :-

public static void main(String[] args)  {
        final Logger logger = LoggingService.getLogger(FileToFileTopology.class.getName());

        try{
        Properties prop =new Properties();
        prop.load(new FileInputStream(args[0]+"/connection.properties"));

        LoggingService.generateAppender("storm_etl",prop, "");
        logger.info("inside main method...." +args.length);

        System.out.println("inside main sys out");

        Config conf= new Config();
        conf.setDebug(false);
        conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING,1);
        TopologyBuilder builder = new TopologyBuilder();

            builder.setSpout("file-reader",new FileReaderSpout(args[1]));
            builder.setBolt("file-writer",new WriteToFileBolt(args[1]),2).shuffleGrouping("file-reader");
            logger.info("submitting topology");
            StormSubmitter.submitTopology(args[0], conf, builder.createTopology());


    }
    catch(Exception e){
        System.out.println("inside catch");
        logger.info("inside catch"+e.getMessage());
        logger.error("inside error", e);
        e.printStackTrace();
    }
    }

我还使用 log4j 为我的拓扑创建了自己的日志文件，创建了日志文件但我的日志文件中没有错误.请帮忙

I have also used log4j to create my own logfile for my topology, log file gets created but no error in my log file. pls help

推荐答案

我在使用 Hortonworks2.2 时遇到了同样的问题.这是因为权限所致.

I had same issue with Hortonworks2.2. This happened because of permissions.

即使您以 Root 用户身份提交到集群，在提交 Storm jar 命令时，它也会以 'storm' 用户身份执行.它可以从源读取文件，但不会写入，因为它没有必要的权限.

Even if you are submitting to the cluster as Root user, when submitting the storm jar command, it executes as 'storm' user. It can read the file from source, but won't write, because it doesn't have the necessary rights.

修改要写入文件的目标文件夹的权限，使用所有权限.

Modify the permissions of destination folder where you want to write file with all permissions.

chmod 777 -R /user/filesfolder

这篇关于使用storm在远程集群中运行拓扑时文件创建问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！