使用mgo对MongoDB进行并发查询以封闭套接字结束 [英] Concurrent queries to MongoDB using mgo ends in closed sockets

问题描述

我一直在检查几个问题在这里stackoverflow +一些职位，人们公开了如何管理会议的例子 mgo golang lib 。

关键是我看到的所有例子并不会同时运行太多的并发查询。增加并发操作的数量在关闭的套接字中结束。在这里你可以找到我为了重现这种行为而运行的代码。

使用mgo结尾的并发查询到MongoDB结束于关闭请注意，我只运行了200个并发查询，为每个查询都打开一个新的套接字。 200不是一个大数字。

我看到的错误是：

阅读tcp 127.0.0.1:59583-> 127.0.0.1:27018：read：连接重置由peer

已关闭明确地

我应该如何处理这个问题？我想保持这个并发级别，甚至可以在某个时候增加它。

解决方案

结果与Mongo数据库相同如果我将插入的数目提高到5 000以上，则不在码头容器中。
没有会话拷贝的插入工作正常，并且带有会话拷贝的插入最终导致mongo错误明确关闭。

根据威廉肯尼迪博客，推荐的方式确实是做会话复制。否则，操作将在同一个套接字中与其他go例程序列化。

如果您计算使用mongoldb打开的套接字数量：
，则为true;做lsof -i | grep 27017 | wc -l;睡觉0.5;完成
，你会看到当我们没有会话拷贝时，我们有一个套接字打开。
使用会话复制时，套接字的数量可以提高数字。

我认为在mongo或系统中达到了打开的套接字限制。可以使用 ulimit 来检查shell环境的限制。

请注意，根据设计，mgo不允许对会话编号进行任何限制。这个想法是避免在数据库访问方面受到限制，并推动开发人员在其代码的入口处引入限制：例如，限制同时处理的http请求的数量。

I have been checking several questions here in stackoverflow + some posts where people exposes examples of of how to manage sessions with mgo golang lib .

The point is that all the examples I've seen doesn't run too much concurrent queries at the same time. Increasing the number of concurrent operations ends in closed sockets. Here you can find the code I ran in order to reproduce this behaviour.

Concurrent queries to MongoDB using mgo ends in closed sockets.

Note that I'm just running 200 of concurrent queries, opening a new socket for each one. 200 is not a big number.

The errors I see are:

read tcp 127.0.0.1:59583->127.0.0.1:27018: read: connection reset by peer

Closed explicitly

How should I deal with this? I want to keep this concurrency level, even increase it at some point.

解决方案

The result is the same with a Mongo database which is not in a docker container if I raise the number of insert to more than 5 000. The inserts with no session copy work ok and the inserts with session copy end up with mongo errors "Closed explicitly".

According to William Kennedy blog, the recommended way is indeed to make session copy. Otherwise, the operations are serialized with other go routines in the same socket.

If you count the number of sockets opened with mongoldb : while true ; do lsof -i | grep 27017 | wc -l ; sleep 0.5 ; done you will see that we have one socket open when we have no session copy. When session copy is used, the number of sockets can raise high number.

I think that an open socket limit is reached in mongo or in the system. ulimit can be used to check the limits of the shell environment.

Note that, by design, mgo doesn't allow any limit on session number. The idea is avoid having limits on the db access side and push the developer to introduce limitations at the entry point of his code: limit the number of http requests treated simultaneously, for instance.

这篇关于使用mgo对MongoDB进行并发查询以封闭套接字结束的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！