使用“storm jar"远程部署 Storm 拓扑Windows 上的命令 [英] Deploy Storm topology remotely using "storm jar" command on Windows
问题描述
我正在尝试在 Windows 上使用storm jar 命令将风暴拓扑部署到远程机器
I am trying to deploy a storm topology to a remote machine using the storm jar command on Windows
storm jar jarName.jar mainClass
这是我的storm.yaml文件:
Here is what I have in my storm.yaml file:
storm.zookeeper.servers:
- "HostName"
nimbus.host: "HostName"
nimbus.thrift.port: 6627
但是每当我尝试部署时，它总是尝试部署到我的本地机器上.
But whenever I try to deploy, it always tries to deploy to my local machine.
编辑
我尝试放置以下配置但它不起作用:
I tried putting the following configurations but it does not work:
conf.put(Config.NIMBUS_HOST, "HostName");
conf.put(Config.NIMBUS_THRIFT_PORT,6627);
conf.put(Config.STORM_ZOOKEEPER_SERVERS,Arrays.asList(new String[]{"HostName"}));
conf.put(Config.STORM_ZOOKEEPER_PORT,2181);
我也在命令行上尝试了 -c 选项:
I also tried the -c option on the command line as:
storm -c nimbus.host=HostName jar jarName.jar mainClass
但是不行；我得到:
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
我尝试将 -c 部分放在不同的位置，但似乎没有任何效果.
I tried placing the -c part at different places but nothing seems to work.
另外，我明白了
The field Config.NIMBUS_HOST is deprecated
当我设置时:
conf.put(Config.NIMBUS_HOST, "HostName");
我使用的是 Storm 版本 0.10.0-beta
I am using Storm version 0.10.0-beta
EDIT2
在我使用后从 IDE 提交工作正常
Submitting works perfectly fine from IDE after I use
System.setProperty("storm.jar", "path to jar");
但是，它不能从命令行运行.
however, it does not work from command line.
编辑 3
感谢马特的所有帮助.我能够使用 eclipse 执行的 java 命令部署拓扑.但是，我仍然无法使用storm jar 命令进行部署.它抱怨拓扑已经存在，即使我在 Storm UI 上没有看到任何东西.此外，当我从命令行使用 Eclipse 或 Java 命令进行部署时，拓扑部署成功.我应该如何比较执行 java 命令和 Storm jar 命令正在执行的操作之间的区别?我试图在 cmd 文件的这一部分中回显变量
Thanks Matt for all your help. I was able to deploy the topology using the java command that eclipse executes. I am still unable to deploy using the storm jar command however. It complains that the topology already exists even though i dont see anything on the Storm UI. Also, when i deploy using Eclipse or the Java command from command line, the topology deploys successfully. How should i compare the difference between executing the java command and what the storm jar command is doing? i tried to echo the variables in this section of the cmd file
if %storm-command% == jar (
set STORM_OPTS=%STORM_CLIENT_OPTS% %STORM_OPTS% -Dstorm.jar=%2
echo %STORM_OPTS%
set CLASSPATH=%CLASSPATH%;%2
echo %CLASSPATH%
set CLASS=%3
set args=%4
goto start
:start
shift
if [%4] == [] goto done
set args=%args% %4
echo args: %args%
goto start
:done
echo %args%
set storm-command-arguments=%args%
)
但我没有看到任何异常.另外，我在堆栈跟踪中看到以下异常
but i dont see anything unusual. Also, i see the following exception in the stacktrace
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
注意
topologies:[]
然后它抱怨拓扑名称存在
and then it complains that the topology name exists
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at mainClass.submitTopology(mainClass.java:127)
at mainClass.main(mainClass.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:613)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:549)
at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16546)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16531)
at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
... 5 more
最后，这里是整个堆栈跟踪.
finally, here is the entire stacktrace.
X:\abc\xyz\target>ERROR StatusLogger Invalid URL C:/asd/ewt/apache-storm 0.10.0-beta/log4j2/cluster.xml java.net.MalformedURLException: unknown protocol: c
at java.net.URL.<init>(URL.java:593)
at java.net.URL.<init>(URL.java:483)
at java.net.URL.<init>(URL.java:432)
at java.net.URI.toURL(URI.java:1089)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getInputFromUri(ConfigurationFactory.java:296)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:395)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:254)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:419)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:138)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:175)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:102)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
at org.apache.log4j.Category.<init>(Category.java:56)
at org.apache.log4j.Logger.<init>(Logger.java:35)
at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:59)
at org.apache.log4j.Logger.getLogger(Logger.java:39)
at mainClass.<clinit>(mainClass.java:32)
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException: Required field nimbus_uptime_secs' is unset!Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at mainClass.submitTopology(mainClass.java:127)
at mainClass.main(mainClass.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:[SupervisorSummary(host:HostName, uptime_secs:76688, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:613)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.
read(ClusterSummary.java:549)
at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16546)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_
resultStandardScheme.read(Nimbus.java:16531)
at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
... 5 more
顺便说一下，堆栈跟踪顶部的那个奇怪的日志记录异常/错误是什么?这可能与storm jar命令的奇怪行为有关吗?
and by the way what is that weird logging exception/error at the top of the stacktrace? could that have something to do with the weird behavior of storm jar command?
如果我这样做
java -cp jarName.jar;C:\abc\asdasd\apache-storm-0.10.0-beta\lib\* mainClass
我明白了
513 [main] INFO b.s.u.Utils - Using defaults.yaml from resources
582 [main] INFO b.s.u.Utils - Using defaults.yaml from resources
590 [main] INFO b.s.StormSubmitter - Generated ZooKeeper secret payload for MD
5-digest: -6100635753051151148:-8346710213172614614
591 [main] INFO b.s.s.a.AuthUtils - Got AutoCreds []
599 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
626 [main] INFO b.s.u.StormBoundedExponentialBackoffRetry - The baseSleepTimeM
s [2000] the maxSleepTimeMs [60000] the maxRetries [5]
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.protoc
ol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName, uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:342)
at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:226)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
at com.rbccm.th2015.storm.SampleTopologyX.submitTopology(SampleTopologyX.java:127)
at com.rbccm.th2015.storm.SampleTopologyX.main(SampleTopologyX.java:57)
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field 'nimbus_uptime_secs' is unset! Struct:ClusterSummary(supervisors:SupervisorSummary(host:HostName,uptime_secs:83775, num_workers:2, num_used_workers:0, supervisor_id:7e4b13c5-05d5-4944-9566-d71f2c2de504, version:0.10.0.2.3.0.0-2557)], nimbus_uptime_secs:0, topologies:[])
at backtype.storm.generated.ClusterSummary.validate(ClusterSummary.java:515)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:613)
at backtype.storm.generated.ClusterSummary$ClusterSummaryStandardScheme.read(ClusterSummary.java:549)
at backtype.storm.generated.ClusterSummary.read(ClusterSummary.java:473)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16546)
at backtype.storm.generated.Nimbus$getClusterInfo_result$getClusterInfo_resultStandardScheme.read(Nimbus.java:16531)
at backtype.storm.generated.Nimbus$getClusterInfo_result.read(Nimbus.java:16470)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
at backtype.storm.generated.Nimbus$Client.recv_getClusterInfo(Nimbus.java:569)
at backtype.storm.generated.Nimbus$Client.getClusterInfo(Nimbus.java:557)
at backtype.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:333)
... 5 more
所以它抱怨同样的事情，但通过不同的途径.这可能与我的 YAML 有关系吗?关于 nimbus_uptime_secs 的业务是什么?
so its complaining about the same thing but via a different route. could this have something to do with my YAML? and what is this business about nimbus_uptime_secs?
最终编辑
好的，是时候放下它了.至少现在看来 sotrm jar 命令不适合 Windows.我可以清楚地看到 Storm.py 尝试在 Windows 中执行的命令与它尝试在 Linux 中执行的命令之间存在差异.基本上，路径在 Windows 中被搞砸了，这一定是导致所有这些奇怪异常的原因.
ok, time to put this to rest. For now at least, it does not look like the sotrm jar command is suitable for windows. I can clearly see a difference in the command that storm.py tries to execute in Windows and the one that it tries to execute in Linux. Basically, the paths get screwed up in windows and that must be what's causing all of these weird exceptions.
感谢马特的所有帮助和支持.我真的学到了很多.
Thanks Matt for all of your help and support. I really learned a lot.
推荐答案
通过 storm jar 运行拓扑仅适用于 Linux(但不适用于 Windows).storm"命令是一个 Python 脚本，假设在 Linux 上运行(Storm 官方不支持 Window).
Running a topology via storm jar does only work on Linux (but not on Windows). The "storm" command is a Python script that assumes to run on Linux (Window is officially not supported by Storm).
然而，storm"命令仅启动具有适当设置的常规 java 命令.如果拓扑是通过 IDE 提交的，IDE 也会做同样的事情.如果您使用 Eclipse，则可以按照 Eclipse:导出运行配置
However, "storm" command only starts a regular java command with appropriate setting. If a topology is submitted via an IDE, the IDE does the same thing. If you use Eclipse, you can extract the used configuration as described in Eclipse: export running configuration
注意，需要设置JVM参数-Dstorm.jar=path\to\myJar.jar；否则，StormSubmitter 将找不到应该提交给 Storm 集群的 jar.
Pay attention, that you need to set JVM argument -Dstorm.jar=path\to\myJar.jar; otherwise, StormSubmitter will not find the jar that should be submitted to Storm cluster.
提取java 命令后，它应该允许您在Windows 上通过命令行提交拓扑.如果您的 IDE 不支持提取命令，您仍然可以手动将其组合在一起(即使这是一个繁琐的解决方案).
After extracting the java command, it should allow you to submit a topology via command line on Windows. If extracting the command is not supported by your IDE, you can still put it together manually (even if this is a cumbersome solution).
它将类似于:
java -Dstorm.jar=yourJarFile.jar
-classpath yourJarFile.jar:
/home/mjsax/.m2/repository/org/apache/storm/storm-core/0.11.0-SNAPSHOT/storm-core-0.11.0-SNAPSHOT.jar:
/home/mjsax/.m2/repository/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar:
/home/mjsax/.m2/repository/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar:
/home/mjsax/.m2/repository/org/ow2/asm/asm/4.0/asm-4.0.jar:
/home/mjsax/.m2/repository/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar:
/home/mjsax/.m2/repository/org/objenesis/objenesis/1.2/objenesis-1.2.jar:
/home/mjsax/.m2/repository/org/clojure/clojure/1.6.0/clojure-1.6.0.jar:
/home/mjsax/.m2/repository/clj-time/clj-time/0.8.0/clj-time-0.8.0.jar:
/home/mjsax/.m2/repository/joda-time/joda-time/2.3/joda-time-2.3.jar:
/home/mjsax/.m2/repository/compojure/compojure/1.1.3/compojure-1.1.3.jar:
/home/mjsax/.m2/repository/org/clojure/core.incubator/0.1.0/core.incubator-0.1.0.jar:
/home/mjsax/.m2/repository/org/clojure/tools.macro/0.1.0/tools.macro-0.1.0.jar:
/home/mjsax/.m2/repository/clout/clout/1.0.1/clout-1.0.1.jar:
/home/mjsax/.m2/repository/ring/ring-core/1.1.5/ring-core-1.1.5.jar:
/home/mjsax/.m2/repository/commons-codec/commons-codec/1.6/commons-codec-1.6.jar:
/home/mjsax/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:
/home/mjsax/.m2/repository/commons-fileupload/commons-fileupload/1.2.1/commons-fileupload-1.2.1.jar:
/home/mjsax/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:
/home/mjsax/.m2/repository/hiccup/hiccup/0.3.6/hiccup-0.3.6.jar:
/home/mjsax/.m2/repository/ring/ring-devel/1.3.0/ring-devel-1.3.0.jar:
/home/mjsax/.m2/repository/clj-stacktrace/clj-stacktrace/0.2.7/clj-stacktrace-0.2.7.jar:
/home/mjsax/.m2/repository/ns-tracker/ns-tracker/0.2.2/ns-tracker-0.2.2.jar:
/home/mjsax/.m2/repository/org/clojure/tools.namespace/0.2.4/tools.namespace-0.2.4.jar:
/home/mjsax/.m2/repository/org/clojure/java.classpath/0.2.2/java.classpath-0.2.2.jar:
/home/mjsax/.m2/repository/ring/ring-jetty-adapter/1.3.0/ring-jetty-adapter-1.3.0.jar:
/home/mjsax/.m2/repository/ring/ring-servlet/1.3.0/ring-servlet-1.3.0.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/jetty-server/7.6.13.v20130916/jetty-server-7.6.13.v20130916.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/orbit/javax.servlet/2.5.0.v201103041518/javax.servlet-2.5.0.v201103041518.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/jetty-continuation/7.6.13.v20130916/jetty-continuation-7.6.13.v20130916.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/jetty-http/7.6.13.v20130916/jetty-http-7.6.13.v20130916.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/jetty-io/7.6.13.v20130916/jetty-io-7.6.13.v20130916.jar:
/home/mjsax/.m2/repository/org/eclipse/jetty/jetty-util/7.6.13.v20130916/jetty-util-7.6.13.v20130916.jar:
/home/mjsax/.m2/repository/ring/ring-json/0.3.1/ring-json-0.3.1.jar:
/home/mjsax/.m2/repository/cheshire/cheshire/5.3.1/cheshire-5.3.1.jar:
/home/mjsax/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.3.1/jackson-core-2.3.1.jar:
/home/mjsax/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.3.1/jackson-dataformat-smile-2.3.1.jar:
/home/mjsax/.m2/repository/tigris/tigris/0.1.1/tigris-0.1.1.jar:
/home/mjsax/.m2/repository/org/clojure/tools.logging/0.2.3/tools.logging-0.2.3.jar:
/home/mjsax/.m2/repository/com/googlecode/disruptor/disruptor/2.10.4/disruptor-2.10.4.jar:
/home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-api/2.1/log4j-api-2.1.jar:
/home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-core/2.1/log4j-core-2.1.jar:
/home/mjsax/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.1/log4j-slf4j-impl-2.1.jar:
/home/mjsax/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar:
/home/mjsax/.m2/repository/org/slf4j/log4j-over-slf4j/1.6.6/log4j-over-slf4j-1.6.6.jar:
/home/mjsax/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:
/home/mjsax/.m2/repository/args4j/args4j/2.0.16/args4j-2.0.16.jar
package.to.your.MainClass
这篇关于使用“storm jar"远程部署 Storm 拓扑Windows 上的命令的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
