用于 Storm 拓扑的 Mongo 连接池 [英] Mongo connection pooling for Storm topology

问题描述

要连接到 mongo 数据库，在我们的非风暴应用程序中，我们将创建一个 Mongo 类的单例实例并在整个应用程序中共享它

想知道在 Storm 框架中实现 Mongo 连接池的正确方法是什么?

我们尝试过的选项:
我们有一个从 Mongo 数据库读取的 spout.我们只是将单例 Mongo 类放在 spout 中并在 open 方法中初始化它?- 但是这种方法使得在需要时无法将 Mongo 实例共享给 Bolt.
想要一些正确的方法.

To connect to a mongo database,in our non-storm applications we would create a singleton instance of the Mongo class and share it across the application

Would like to know what is the right way to implement a Mongo connection pooling in the storm framework?

Options we have tried :
We have a spout which reads from the Mongo database. we just put the singleton Mongo class inside the spout and initialise it in the open method? - But this approach made it impossible to share the Mongo instance to a bolt if needed.
would like to have some pointers to right way of doing it.

推荐答案

storm 中的连接池有点复杂，因为您需要考虑以下几点:

Connection pooling within storm is a little bit more complicated since you need to consider a couple things:

1. 您在集群环境中运行 -> 多台机器需要自己的连接
2. 每个 Storm worker 在单独的 JVM 中运行 -> 执行器很可能不在同一个 worker 中运行......这意味着没有共享连接

1. You are running in a clustered environment -> multiple machines need their own connections
2. Each Storm worker runs in a separate JVM -> it is likely that executors are not running in the same worker... which means no sharing connections

所以，不要太担心.继续像往常一样在 Spouts 和 Bolts 中使用类的 Singleton 实例(如@bridiver 的回答).如果它们碰巧共享同一个 JVM，那么它们将共享同一个池，你就可以开始了！

So, don't worry too much about it. Continue to use the Singleton instance (like in @bridiver's answer) of your class within your Spouts and Bolts like normal. If they happen to share the same JVM, then they will share the same pool and you are good to go!

关于初始化，这是我所做的:

In regards to the initialization, here is what I do:

1. 通过构造函数将配置信息传递到 spout/bolt 中
2. 在open(对于spouts)或prepare(对于bolts)方法中初始化你的连接
3. 连接在喷口和螺栓之间不共享

1. Pass configuration information into the spout/bolt via the constructor
2. Initialize your connections in the open (for spouts) or the prepare (for bolts) methods
3. Connections are not shared between the spout and bolt

这篇关于用于 Storm 拓扑的 Mongo 连接池的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！