将套接字对象发送到分叉的运行进程(multiprocessing.Queue) [英] Send socket object to forked running process (multiprocessing.Queue)

问题描述

我正在学习使用 HTML5 WebSockets，作为其中的一部分，我正在用 Python 编写一个服务器，以便我了解它们的工作原理.前几天我创建了一个效果很好的，但我想扩展它，以便它支持多个端点，每个端点都是一个不同的服务"，可以处理 websocket 客户端.

I am learning to use HTML5 WebSockets and as part of that I am writing a server in Python so I can know the nitty gritty of how they work. I created one the other day that worked pretty well, but I wanted to expand it so it would support multiple endpoints with each endpoint being a different "service" which can handle websocket clients.

目前，我的实现适用于生成进程等(我使用多处理而不是线程，因为我读到线程在 CPython 中并不是真正的多线程，这就是我认为我正在使用的(默认安装在 Ubuntu 12.04 上))，但我无法将接收到的客户端套接字发送到服务进程.

At the moment, my implementation works with spawning processes and such (I am using multiprocessing instead of threading since I read that threading isn't really multithreading in CPython and that's what I think I am using (default install on Ubuntu 12.04)), but I am having trouble sending received client sockets to the service processes.

这是我发送它们的方式(循环运行):

Here is how I send them (this runs in a loop):

try:
    #get a new client
    conn, addr = server.accept()
    print "Client connected from", addr
    request = conn.recv(4096)
    response, close, service = self.handshake(request)
    conn.send(response)
    if close:
        print "Invalid request from", addr
        conn.close()
        continue
    client = WebSockets.WebSocketClient(conn, addr)
    service.clientConnQueue.put(client)

'server' 是一个监听传入连接的套接字.握手方法负责验证他们的请求并确定将客户端放入哪个服务进程.'response' 是要发送的 http 响应，如果有错误，'close' 是 True，'service' 是一个继承自 multiprocessing.Queue 的类.WebSocktes.WebSocketClient 是存储我的发送和接收实现的地方，它基本上充当套接字的包装器.

'server' is a socket listening for incoming connections. The handshake method takes care of validating their request and determining which service process to put the client into. 'response' is the http response to send, 'close' is True if there was an error, and 'service' is a class which inherits from multiprocessing.Queue. WebSocktes.WebSocketClient is where my sending and receiving implementation is stored and it basically functions as a wrapper for a socket.

'clientConnQueue' 是这样创建的:

'clientConnQueue' is created like so:

class Service(multiprocessing.Process):
    """Base class for all services."""
    def __init__(self, manager):
        multiprocessing.Process.__init__(self)
        self.manager = manager
        self.clientConnQueue = self.manager.Queue()
        self.shutdownFlag = multiprocessing.Event()

manager 是一个 multiprocessing.Manager()

manager is a multiprocessing.Manager()

尝试将客户端放入 clientConnQueue 时出现的错误如下:

The error I get when I try to place a client in the clientConnQueue is as follows:

File "./WebSocketServer.py", line 183, in <module>
main()
File "./WebSocketServer.py", line 180, in main
server.runServer()
File "./WebSocketServer.py", line 67, in runServer
service.clientConnQueue.put(client)
File "<string>", line 2, in put
File "/usr/lib/python2.7/multiprocessing/managers.py", line 758, in _callmethod
conn.send((self._id, methodname, args, kwds))
TypeError: expected string or Unicode object, NoneType found

然后我在接收端遇到了管道损坏错误.

I then get a broken pipe error on the receiving side.

我在使用 multiprocessing.Queue 发送连接时遇到了同样的错误，我认为将其更改为由管理器创建的队列可以解决该问题.然而，它似乎做了完全相同的实现.

I got the same error when I was using a multiprocessing.Queue to send the connection and I thought that changing it to a queue created by a manager would fix the problem. However, it seems to do the exact same implementation.

显然，这不是向正在运行的进程发送类似内容的方式，那么将不可序列化的对象发送到进程的正确方法是什么?

Clearly this isn't the way one is supposed to send something like this to a running process, so what is the proper way to be sending non-serializable objects to a process?

推荐答案

将套接字传递给另一个进程并不是一件小事.看，例如，这个问题:我可以打开一个套接字并将它传递给 Linux 中的另一个进程

Pass socket to another process is not a trivial thing. Look, for example, this question: Can I open a socket and pass it to another process in Linux

无论如何，由于大量的内存开销，操作系统进程或线程并不是您真正想要实现 websocket 服务器的.看看非阻塞套接字的 smth ......例如，Tornado http://www.tornadoweb.org/文档/websocket.html

Anyway, OS processes or threads is not what you really want for implement websocket server, because of large memory overhead. Look at smth with nonblocking sockets... eg, Tornado http://www.tornadoweb.org/documentation/websocket.html

这篇关于将套接字对象发送到分叉的运行进程(multiprocessing.Queue)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！