Storm拓扑配置 [英] Storm topology configuration

问题描述

您如何为风暴拓扑提供自定义配置?例如，如果我构建了一个连接到 MySQL 集群的拓扑，并且我希望能够在不重新编译的情况下更改我需要连接的服务器，我该怎么做?我更喜欢使用配置文件，但我担心该文件本身并未部署到集群中，因此不会运行(除非我对集群工作方式的理解存在缺陷).到目前为止，我看到的在运行时将配置选项传递到 Storm 拓扑的唯一方法是通过命令行参数，但是当您获得大量参数时，这会很麻烦.

How do you provide a custom configuration to a storm topology? For example, if I have a topology that I built that connects to a MySQL cluster and I want to be able to change which servers I need to to connect to without recompiling, how would I do that? My preference would be to use a config file, but my concern is that the file itself is not deployed to the cluster, therefore it won't be run (unless my understanding of how a cluster works is flawed). The only way I've seen so far to pass configuration options into a storm topology at runtime is via a command-line parameter, but that is messy when you get a good number of parameters.

确实有一个想法是利用 shell 脚本将文件读入一个变量并将该变量的内容作为字符串传递给拓扑，但如果可能的话，我想要更简洁的东西.

One thought did have is to leverage a shell script to read the file into a variable and pass the contents of that variable in as a string to the topology, but I'd like something a little cleaner if possible.

有没有其他人遇到过这种情况?如果是，你是怎么解决的?

Has anyone else encountered this? If so, how did you solve it?

似乎需要提供更多说明.我的场景是我有一个拓扑，我希望能够在不同的环境中部署它而无需重新编译它.通常，我会创建一个配置文件，其中包含数据库连接参数之类的内容并将其传入.我想知道如何在 Storm 中执行类似的操作.

It appears to need to provide more clarification. My scenario is that I have a topology that I want to be able to deploy in different environments without having to recompile it. Normally, I'd create a config file that contains things like database connection parameters and have that passed in. I'd like to know how to do something like that in Storm.

推荐答案

您可以指定与拓扑一起提交的配置(通常通过 yaml 文件).我们如何在我们自己的项目中自己管理这个是我们有单独的配置文件用于开发和一个用于生产，在其中我们存储我们的服务器、redis 和数据库 IP 和端口等.然后当我们运行我们的命令来构建 jar 并提交时风暴拓扑包括正确的配置文件，具体取决于您的部署环境.Bolts 和 Spouts 只是从 StormConf 映射中读取它们需要的配置，该映射在您的 bolt 的 prepare() 方法中传递给它们.

You can specify a configuration (via a yaml file typically) which you submit with your topology. How we manage this ourselves in our own project is we have separate config files for development and one for production, and inside it we store our server, redis and db IPs and Ports etc. Then when we run our command to build the jar and submit the topology to storm it includes the correct config file depending on your deployment environment. The bolts and spouts simply read the configuration they require from the stormConf map which is passed to them in your bolt's prepare() method.

来自 http://storm.apache.org/documentation/Configuration.html:

每个配置都有一个默认值定义在 Storm 代码库的 defaults.yaml 中.你可以通过在 Nimbus 和 supervisor 的类路径中定义一个storm.yaml 来覆盖这些配置.最后，您可以定义在使用 StormSubmitter 时与拓扑一起提交的特定于拓扑的配置.但是，特定于拓扑的配置只能覆盖以TOPOLOGY"为前缀的配置.

Every configuration has a default value defined in defaults.yaml in the Storm codebase. You can override these configurations by defining a storm.yaml in the classpath of Nimbus and the supervisors. Finally, you can define a topology-specific configuration that you submit along with your topology when using StormSubmitter. However, the topology-specific configuration can only override configs prefixed with "TOPOLOGY".

Storm 0.7.0 及更高版本允许您基于每个螺栓/每个喷嘴覆盖配置.

Storm 0.7.0 and onwards lets you override configuration on a per-bolt/per-spout basis.

您还将在 http://nathanmarz.github 上看到.io/storm/doc/backtype/storm/StormSubmitter.html 向 submitJar 和 submitTopology 传递了一个名为 conf 的映射.

You'll also see on http://nathanmarz.github.io/storm/doc/backtype/storm/StormSubmitter.html that submitJar and submitTopology is passed a map called conf.

希望这能帮助您入门.

这篇关于Storm拓扑配置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！