在 Storm 0.10.0 上，即使我设置了 worker=1，也会启动 2 个工作进程，并且 UI 报告 worker=1 [英] on Storm 0.10.0 2 worker processes are launched even when I set workers=1, and UI reports that workers=1
问题描述
我有一个风暴拓扑:setNumWorkers(1);
当我查看有关此正在运行的拓扑的 Storm UI 报告时，我看到 Num workers 设置为 1.
When I look at the storm UI report on this running topology, I see Num workers set to 1.
但是，当我登录到运行主管的节点时，我看到两个进程对 -Dworker.id 和 -Dworker.port 具有相同的设置.
我在下面包含了 ps 向我展示的这两个过程的输出.
However, when i log into the node running the supervisor I see two processes that have the same setting for -Dworker.id and for -Dworker.port.
I am including the output of what ps shows me for these two processes below.
我的问题是:如果我只请求一个，为什么有两个进程似乎被配置为工作进程(注意:Storm UI 确认我只有一个工作进程.)
My question is: Why are there two processes that seem to be configured as worker processes if I only requested one (note: the Storm UI confirms that I have only one worker.)
这对我很重要，因为当我对拓扑消耗的资源进行任何剖析或分析时，我想知道将哪个进程归零.
This is important to me because when I do any profiling or analysis of what resources are being consumed by my topology, I want to know which process to zero in on.
ps 输出
root 787 20.0 0.6 5858228 78388 ? Sl 05:04 0:00 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /opt/apache-storm-0.10.0/lib/log4j-slf4j-impl-2.1.jar:/opt/apache-storm-0.10.0/lib/servlet-api-2.5.jar:/opt/apache-storm-0.10.0/lib/clojure-1.6.0.jar:/opt/apache-storm-0.10.0/lib/slf4j-api-1.7.7.jar:/opt/apache-storm-0.10.0/lib/hadoop-auth-2.4.0.jar:/opt/apache-storm-0.10.0/lib/log4j-api-2.1.jar:/opt/apache-storm-0.10.0/lib/disruptor-2.10.4.jar:/opt/apache-storm-0.10.0/lib/storm-core-0.10.0.jar:/opt/apache-storm-0.10.0/lib/log4j-over-slf4j-1.6.6.jar:/opt/apache-storm-0.10.0/lib/log4j-core-2.1.jar:/opt/apache-storm-0.10.0/lib/asm-4.0.jar:/opt/apache-storm-0.10.0/lib/kryo-2.21.jar:/opt/apache-storm-0.10.0/lib/reflectasm-1.07-shaded.jar:/opt/apache-storm-0.10.0/lib/minlog-1.2.jar:/opt/apache-storm-0.10.0/conf:/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/stormjar.jar -Dlogfile.name=big-storm-job-1-1487739502-worker-6700.log -Dstorm.home=/opt/apache-storm-0.10.0 -Dstorm.id=big-storm-job-1-1487739502 -Dworker.id=e8e03e95-1fcc-492a-b5e4-51ef7b8db2ee -Dworker.port=6700 -Dstorm.log.dir=/opt/apache-storm-0.10.0/logs -Dlog4j.configurationFile=/opt/apache-storm-0.10.0/log4j2/worker.xml backtype.storm.LogWriter /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx768m -Djava.library.path=/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/resources/Linux-amd64:/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=big-storm-job-1-1487739502-worker-6700.log -Dstorm.home=/opt/apache-storm-0.10.0 -Dstorm.conf.file= -Dstorm.options= -Dstorm.log.dir=/opt/apache-storm-0.10.0/logs -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/opt/apache-storm-0.10.0/log4j2/worker.xml -Dstorm.id=big-storm-job-1-1487739502 -Dworker.id=e8e03e95-1fcc-492a-b5e4-51ef7b8db2ee -Dworker.port=6700 -cp /opt/apache-storm-0.10.0/lib/log4j-slf4j-impl-2.1.jar:/opt/apache-storm-0.10.0/lib/servlet-api-2.5.jar:/opt/apache-storm-0.10.0/lib/clojure-1.6.0.jar:/opt/apache-storm-0.10.0/lib/slf4j-api-1.7.7.jar:/opt/apache-storm-0.10.0/lib/hadoop-auth-2.4.0.jar:/opt/apache-storm-0.10.0/lib/log4j-api-2.1.jar:/opt/apache-storm-0.10.0/lib/disruptor-2.10.4.jar:/opt/apache-storm-0.10.0/lib/storm-core-0.10.0.jar:/opt/apache-storm-0.10.0/lib/log4j-over-slf4j-1.6.6.jar:/opt/apache-storm-0.10.0/lib/log4j-core-2.1.jar:/opt/apache-storm-0.10.0/lib/asm-4.0.jar:/opt/apache-storm-0.10.0/lib/kryo-2.21.jar:/opt/apache-storm-0.10.0/lib/reflectasm-1.07-shaded.jar:/opt/apache-storm-0.10.0/lib/minlog-1.2.jar:/opt/apache-storm-0.10.0/conf:/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/stormjar.jar backtype.storm.daemon.worker big-storm-job-1-1487739502 8fde2226-4b32-406d-8809-81ed88e5ae1f 6700 e8e03e95-1fcc-492a-b5e4-51ef7b8db2ee
root 805 203 2.0 4308648 255336 ? Sl 05:04 0:06 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -server -Xmx768m -Djava.library.path=/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/resources/Linux-amd64:/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=big-storm-job-1-1487739502-worker-6700.log -Dstorm.home=/opt/apache-storm-0.10.0 -Dstorm.conf.file= -Dstorm.options= -Dstorm.log.dir=/opt/apache-storm-0.10.0/logs -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/opt/apache-storm-0.10.0/log4j2/worker.xml -Dstorm.id=big-storm-job-1-1487739502 -Dworker.id=e8e03e95-1fcc-492a-b5e4-51ef7b8db2ee -Dworker.port=6700 -cp /opt/apache-storm-0.10.0/lib/log4j-slf4j-impl-2.1.jar:/opt/apache-storm-0.10.0/lib/servlet-api-2.5.jar:/opt/apache-storm-0.10.0/lib/clojure-1.6.0.jar:/opt/apache-storm-0.10.0/lib/slf4j-api-1.7.7.jar:/opt/apache-storm-0.10.0/lib/hadoop-auth-2.4.0.jar:/opt/apache-storm-0.10.0/lib/log4j-api-2.1.jar:/opt/apache-storm-0.10.0/lib/disruptor-2.10.4.jar:/opt/apache-storm-0.10.0/lib/storm-core-0.10.0.jar:/opt/apache-storm-0.10.0/lib/log4j-over-slf4j-1.6.6.jar:/opt/apache-storm-0.10.0/lib/log4j-core-2.1.jar:/opt/apache-storm-0.10.0/lib/asm-4.0.jar:/opt/apache-storm-0.10.0/lib/kryo-2.21.jar:/opt/apache-storm-0.10.0/lib/reflectasm-1.07-shaded.jar:/opt/apache-storm-0.10.0/lib/minlog-1.2.jar:/opt/apache-storm-0.10.0/conf:/opt/apache-storm-0.10.0/storm-local/supervisor/stormdist/big-storm-job-1-1487739502/stormjar.jar backtype.storm.daemon.worker big-storm-job-1-1487739502 8fde2226-4b32-406d-8809-81ed88e5ae1f 6700 e8e03e95-1fcc-492a-b5e4-51ef7b8db2ee
为了防止有人阅读这篇文章以更好地了解我的环境有帮助，这里是我的 Storm(和其他东西)的 docker 配置.希望这会有所帮助.
Just in case it is helpful for someone reading this trying to get a better picture of my environment, here is my docker configuration for Storm (and other stuff). Hopefully that helps.
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
container_name: zk
hostname: zk
ports:
- "2181:2181"
networks:
storm:
kafka:
image: wurstmeister/kafka:0.8.2.2-1
container_name: kafka
hostname: kafka
ports:
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_HOST_NAME: 10.211.55.4
KAFKA_ZOOKEEPER_CONNECT: 10.211.55.4
volumes:
- /var/run/docker.sock:/var/run/docker.sock
nimbus:
image: sunside/storm-nimbus
container_name: storm-nimbus
hostname: storm-nimbus
ports:
- "49773:49772"
- "49772:49773"
- "49627:49627"
environment:
- "LOCAL_HOSTNAME=nimbus"
- "ZOOKEEPER_ADDRESS=zk"
- "ZOOKEEPER_PORT=2181"
- "NIMBUS_ADDRESS=nimbus"
- "NIMBUS_THRIFT_PORT=49627"
- "DRPC_PORT=49772"
- "DRPCI_PORT=49773"
volumes:
- /media/psf/Home/dev/storm-pipeline:/pipeline
networks:
storm:
supervisor:
image: sunside/storm-supervisor
container_name: storm-supervisor
hostname: storm-supervisor
ports:
- "8000:8000"
environment:
- "LOCAL_HOSTNAME=supervisor"
- "NIMBUS_ADDRESS=nimbus"
- "NIMBUS_THRIFT_PORT=49627"
- "DRPC_PORT=49772"
- "DRPCI_PORT=49773"
- "ZOOKEEPER_ADDRESS=zk"
- "ZOOKEEPER_PORT=2181"
networks:
storm:
ui:
image: sunside/storm-ui
container_name: storm-ui
hostname: storm-ui
ports:
- "8888:8080"
environment:
- "LOCAL_HOSTNAME=ui"
- "NIMBUS_ADDRESS=nimbus"
- "NIMBUS_THRIFT_PORT=49627"
- "DRPC_PORT=49772"
- "DPRCI_PORT=49773"
- "ZOOKEEPER_ADDRESS=zk"
- "ZOOKEEPER_PORT=2181"
networks:
storm:
elasticsearch:
image: elasticsearch:2.3
container_name: elasticsearch
hostname: elasticsearch
ports:
- "9200:9200"
networks:
storm:
networks:
storm:
external: true
推荐答案
谜底的答案是，两者的进程之一是真正的工人"进程(正在执行的类是backtype.storm.daemon.worker)... 响应ps"命令而打印出来的另一个进程是一个日志写入进程，由类 backtype.storm.LogWriter 执行.
The answer to the mystery is that one of the process of the two is a real 'worker' process (the class that is being executed is backtype.storm.daemon.worker)... The other process that was printed out in response to the 'ps' command was a log writer process, executed by the class backtype.storm.LogWriter.
我应该在两个进程的输出行中注意到这一点.哦，好吧……现在我们知道了！
I should have noticed this in the output lines for the two processes. Oh well.... now we know !
这篇关于在 Storm 0.10.0 上，即使我设置了 worker=1，也会启动 2 个工作进程，并且 UI 报告 worker=1的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
