urllib.request连接到HTTP服务器的持久性 [英] Persistence of urllib.request connections to a HTTP server

问题描述

我想在我们的一个Web服务器上进行一些性能测试，以了解服务器如何处理大量持久连接。不幸的是，我对HTTP和Web测试并不十分熟悉。这是我到目前为止获得的Python代码：

I want to do some performance testing on one of our web servers, to see how the server handles a lot of persistent connections. Unfortunately, I'm not terribly familiar with HTTP and web testing. Here's the Python code I've got for this so far:

import http.client
import argparse
import threading


def make_http_connection():
    conn = http.client.HTTPConnection(options.server, timeout=30)
    conn.connect()


if __name__ == '__main__':
    parser = argparse.ArgumentParser()

    parser.add_argument("num", type=int, help="Number of connections to make (integer)")
    parser.add_argument("server", type=str, help="Server and port to connect to. Do not prepend \'http://\' for this")

    options = parser.parse_args()

    for n in range(options.num):
        connThread = threading.Thread(target = make_http_connection, args = ())
        connThread.daemon = True
        connThread.start()

    while True:
        try:
            pass
        except KeyboardInterrupt:
            break

我的主要问题是：我如何保持这些连接存活？我设置了一个很长的超时，但这是一个非常粗略的方法我甚至不确定它会影响连接。只需要每隔一段时间请一个或两个字节吗？

My main question is this: How do I keep these connections alive? I've set a long timeout, but that's a very crude method and I'm not even sure it affects the connection. Would simply requesting a byte or two every once in a while do it?

（另外，在一个不相关的注释中，有一个更好的程序等待键盘中断比丑陋的而True：阻止我的代码结尾？）

(Also, on an unrelated note, is there a better procedure for waiting for a keyboard interrupt than the ugly while True: block at the end of my code?)

推荐答案

urllib.request 不支持持久连接。代码中有'Connection：close'硬编码。但是 http.client 部分支持持久连接（包括遗留的http / 1.0 keep-alive ）。因此问题标题可能会产生误导。

urllib.request doesn't support persistent connections. There is 'Connection: close' hardcoded in the code. But http.client partially supports persistent connections (including legacy http/1.0 keep-alive). So the question title might be misleading.

我想在一个上做一些性能测试我们的Web服务器，看看服务器如何处理大量持久连接。不幸的是，我对HTTP和Web测试并不十分熟悉。

I want to do some performance testing on one of our web servers, to see how the server handles a lot of persistent connections. Unfortunately, I'm not terribly familiar with HTTP and web testing.

您可以使用现有的http测试工具，例如 slowloris ，httperf而不是自己写一个。

You could use an existing http testing tools such as slowloris, httperf instead of writing one yourself.

如何保持这些连接存活？

How do I keep these connections alive?

要关闭http / 1.1连接，客户端应显式指定 Connection：close header否则连接被认为是持久的服务器（虽然它可能随时关闭它并且 http.client 将不知道它直到它尝试读/写连接为止。

To close http/1.1 connection a client should explicitly specify Connection: close header otherwise the connection is considered persistent by the server (though it may close it at any moment and http.client won't know about it until it tries to read/write to the connection).

conn.connect（）几乎立即返回，你的线程结束。要强制每个线程维持与服务器的http连接，您可以：

conn.connect() returns almost immediately and your thread ends. To force each thread to maintain an http connection to the server you could:

import time

def make_http_connection(*args, **kwargs):
    while True: # make new http connections
        h = http.client.HTTPConnection(*args, **kwargs)
        while True: # make multiple requests using a single connection
            try:
                h.request('GET', '/') # send request; make conn. on the first run
                response = h.getresponse()
                while True: # read response slooowly
                    b = response.read(1) # read 1 byte
                    if not b:
                       break
                    time.sleep(60) # wait a minute before reading next byte
                    #note: the whole minute might pass before we notice that 
                    #  the server has closed the connection already
            except Exception:
                break # make new connection on any error

注意：如果服务器返回'连接：关闭'则每个连接只有一个请求。

Note: if the server returns 'Connection: close' then there is a single request per connection.

（另外，在一个不相关的注释中，有一个更好的程序等待键盘中断而不是丑陋而True：阻塞在我的代码末尾？）

(Also, on an unrelated note, is there a better procedure for waiting for a keyboard interrupt than the ugly while True: block at the end of my code?)

要等到所有线程f发生inish或 KeyboardInterrupt 你可以：

To wait until all threads finish or KeyboardInterrupt happens you could:

while threads:
    try:
        for t in threads[:]: # enumerate threads
            t.join(.1) # timeout 0.1 seconds
            if not t.is_alive():
               threads.remove(t)
    except KeyboardInterrupt:
        break

或类似的东西：

while threading.active_count() > 1:
    try:
        main_thread = threading.current_thread()
        for t in threading.enumerate(): # enumerate all alive threads
            if t is not main_thread:
               t.join(.1)
    except KeyboardInterrupt:
        break

稍后可能由于各种原因而无法工作，例如，如果存在虚拟线程，例如线程在C扩展中启动而不使用 threading 模块。

The later might not work for various reasons e.g., if there are dummy threads such as threads that started in C extensions without using threading module.

concurrent.futures.ThreadPoolExecutor 提供更高的抽象级别高于线程模块，它可以隐藏一些复杂性。

concurrent.futures.ThreadPoolExecutor provides a higher abstraction level than threading module and it can hide some complexity.

您可以打开多个连接而不是每个连接模型的线程同时在一个线程中，例如，直接使用 requests.async 或 gevent 。

Instead of thread per connection model you could open multiple connections concurrently in a single thread e.g., using requests.async or gevent directly.

这篇关于urllib.request连接到HTTP服务器的持久性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！