管理多个 Twisted 客户端连接 [英] Managing multiple Twisted client connections

问题描述

我正在尝试在一种管理多个客户端连接的爬虫程序中使用 Twisted.我想同时维护大约 5 个客户的池.每个客户端的功能是连接到它从列表中获取的指定 IRC 服务器，进入特定频道，然后将该频道中的用户列表保存到数据库中.

I'm trying to use Twisted in a sort of spidering program that manages multiple client connections. I'd like to maintain of a pool of about 5 clients working at one time. The functionality of each client is to connect to a specified IRC server that it gets from a list, enter a specific channel, and then save the list of the users in that channel to a database.

我遇到的问题比任何事情都更具架构性.我对 Twisted 还很陌生，我不知道有哪些选项可用于管理多个客户端.我假设最简单的方法是简单地让每个 ClientCreator 实例在完成其工作后死亡，并有一个中央循环来检查是否有空间添加新客户端.我认为这不是一个特别不寻常的问题，所以我希望从其他人的经验中收集一些信息.

The problem I'm having is more architectural than anything. I'm fairly new to Twisted and I don't know what options are available for managing multiple clients. I'm assuming the easiest way is to simply have each ClientCreator instance die off once it's completed its work and have a central loop that can check to see if there's room to add a new client. I would think this isn't a particularly unusual problem so I'm hoping to glean some information from other peoples' experiences.

推荐答案

最好的选择实际上只是在这里做显而易见的事情.没有循环或重复的定时调用；只是有做正确事情的处理程序.

The best option is really just to do the obvious thing here. Don't have a loop, or a repeating timed call; just have handlers that do the right thing.

保留一个中央连接管理对象，并使事件处理方法为其提供继续运行所需的信息.启动时，进行 5 个传出连接.跟踪有多少正在进行中，并在其中维护一个列表.当连接成功时(在 connectionMade 中)更新列表以记住连接的新状态.当连接完成时(在 connectionLost 中)告诉连接管理器；它的响应应该是删除该连接并在其他地方建立新连接.在中间，很明显如何触发对您需要的名称的请求并将它们填充到数据库中(等待数据库插入完成，然后断开 IRC 连接，很可能，通过等待 推迟从adbapi回来.

Keep a central connection-management object around, and make event-handling methods feed it the information it needs to keep going. When it starts, make 5 outgoing connections. Keep track of how many are in progress, maintain a list with them in it. When a connection succeeds (in connectionMade) update the list to remember the connection's new state. When a connection completes (in connectionLost) tell the connection manager; its response should be to remove that connection and make a new connection somewhere else. In the middle, it should be fairly obvious how to fire off a request for the names you need and stuff them into a database (waiting for the database insert to complete before dropping your IRC connection, most likely, by waiting for the Deferred to come back from adbapi).

这篇关于管理多个 Twisted 客户端连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！