用于Storm拓扑的Mongo连接池 [英] Mongo connection pooling for Storm topology

问题描述

要连接到mongo数据库，在非风暴应用程序中，我们将创建Mongo类的单例实例，并在应用程序中共享它.

是否想知道在Storm框架中实现Mongo连接池的正确方法是什么?

我们尝试过的选项:
我们有一个从Mongo数据库读取的喷口.我们只是将单例Mongo类放入喷口中，并在open方法中对其进行初始化? -但是这种方法使得无法在需要时将Mongo实例共享给螺栓.
希望有一些正确的方法.

To connect to a mongo database,in our non-storm applications we would create a singleton instance of the Mongo class and share it across the application

Would like to know what is the right way to implement a Mongo connection pooling in the storm framework?

Options we have tried :
We have a spout which reads from the Mongo database. we just put the singleton Mongo class inside the spout and initialise it in the open method? - But this approach made it impossible to share the Mongo instance to a bolt if needed.
would like to have some pointers to right way of doing it.

推荐答案

风暴中的连接池更加复杂，因为您需要考虑以下几点:

Connection pooling within storm is a little bit more complicated since you need to consider a couple things:

1. 您正在集群环境中运行->多台计算机需要自己的连接
2. 每个Storm worker 在单独的JVM中运行->执行程序可能不在同一worker中运行...这意味着没有共享连接

1. You are running in a clustered environment -> multiple machines need their own connections
2. Each Storm worker runs in a separate JVM -> it is likely that executors are not running in the same worker... which means no sharing connections

因此，不必对此太担心.像往常一样，继续在Spouts和Bolts中使用类的Singleton实例(例如@bridiver的答案).如果它们碰巧共享同一个JVM，那么它们将共享同一个池，您就可以了！

So, don't worry too much about it. Continue to use the Singleton instance (like in @bridiver's answer) of your class within your Spouts and Bolts like normal. If they happen to share the same JVM, then they will share the same pool and you are good to go!

关于初始化，这是我的工作:

In regards to the initialization, here is what I do:

1. 通过构造函数将配置信息传递到喷口/螺栓
2. 使用open(用于喷口)或prepare(用于螺栓)方法初始化连接
3. 喷嘴与螺栓之间不共享连接

1. Pass configuration information into the spout/bolt via the constructor
2. Initialize your connections in the open (for spouts) or the prepare (for bolts) methods
3. Connections are not shared between the spout and bolt

这篇关于用于Storm拓扑的Mongo连接池的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！