风暴拓扑配置 [英] Storm topology configuration

问题描述

如何为风暴拓扑提供自定义配置?例如，如果我建立了一个连接到MySQL群集的拓扑，并且希望能够更改需要连接到的服务器而无需重新编译，那么该怎么做?我更喜欢使用配置文件，但我担心的是文件本身未部署到集群，因此将无法运行(除非我对集群的工作原理有一定的了解).到目前为止，我看到的唯一一种在运行时将配置选项传递到Storm拓扑中的方法是通过命令行参数，但是当您获得大量参数时，这很麻烦.

How do you provide a custom configuration to a storm topology? For example, if I have a topology that I built that connects to a MySQL cluster and I want to be able to change which servers I need to to connect to without recompiling, how would I do that? My preference would be to use a config file, but my concern is that the file itself is not deployed to the cluster, therefore it won't be run (unless my understanding of how a cluster works is flawed). The only way I've seen so far to pass configuration options into a storm topology at runtime is via a command-line parameter, but that is messy when you get a good number of parameters.

曾经有一种想法是利用shell脚本将文件读入一个变量并将该变量的内容作为字符串传递给拓扑，但是我希望有一些清洁的方法.

One thought did have is to leverage a shell script to read the file into a variable and pass the contents of that variable in as a string to the topology, but I'd like something a little cleaner if possible.

还有其他人遇到过吗?如果是这样，您是如何解决的?

Has anyone else encountered this? If so, how did you solve it?

似乎需要提供更多说明.我的情况是我有一个拓扑，希望能够在不同的环境中进行部署而不必重新编译它.通常，我将创建一个配置文件，其中包含诸如数据库连接参数之类的内容，并将其传递给我.我想知道如何在Storm中进行类似的操作.

It appears to need to provide more clarification. My scenario is that I have a topology that I want to be able to deploy in different environments without having to recompile it. Normally, I'd create a config file that contains things like database connection parameters and have that passed in. I'd like to know how to do something like that in Storm.

推荐答案

您可以指定要随拓扑提交的配置(通常通过yaml文件).我们如何在自己的项目中自行管理此配置，是我们有单独的配置文件用于开发，一个用于生产，并且在其中存储我们的服务器，redis和db IP和Ports等.然后，当我们运行命令来构建jar并提交时要部署的拓扑包括正确的配置文件，具体取决于您的部署环境.螺栓和喷口只需从stormConf映射中读取所需的配置，该配置将通过螺栓的prepare()方法传递给螺栓和喷口.

You can specify a configuration (via a yaml file typically) which you submit with your topology. How we manage this ourselves in our own project is we have separate config files for development and one for production, and inside it we store our server, redis and db IPs and Ports etc. Then when we run our command to build the jar and submit the topology to storm it includes the correct config file depending on your deployment environment. The bolts and spouts simply read the configuration they require from the stormConf map which is passed to them in your bolt's prepare() method.

来自 http://storm.apache.org/documentation/Configuration.html :

每个配置都有一个默认值，该值在Storm代码库的defaults.yaml中定义.您可以通过在Nimbus和主管的类路径中定义一个storm.yaml来覆盖这些配置.最后，您可以定义使用StormSubmitter时随拓扑一起提交的特定于拓扑的配置.但是，特定于拓扑的配置只能覆盖以"TOPOLOGY"为前缀的配置.

Every configuration has a default value defined in defaults.yaml in the Storm codebase. You can override these configurations by defining a storm.yaml in the classpath of Nimbus and the supervisors. Finally, you can define a topology-specific configuration that you submit along with your topology when using StormSubmitter. However, the topology-specific configuration can only override configs prefixed with "TOPOLOGY".

Storm 0.7.0及更高版本允许您基于每个螺栓/每个喷口来覆盖配置.

Storm 0.7.0 and onwards lets you override configuration on a per-bolt/per-spout basis.

您还将在 http://nathanmarz.github上看到. io/storm/doc/backtype/storm/StormSubmitter.html 会将SubmitJar和SubmitTopology传递给名为conf的地图.

You'll also see on http://nathanmarz.github.io/storm/doc/backtype/storm/StormSubmitter.html that submitJar and submitTopology is passed a map called conf.

希望这可以帮助您入门.

Hope this gets you started.

这篇关于风暴拓扑配置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！