urllib.request连接到HTTP服务器的持久性 [英] Persistence of urllib.request connections to a HTTP server
问题描述
我想在我们的一个Web服务器上进行一些性能测试,以了解服务器如何处理大量持久连接。不幸的是,我对HTTP和Web测试并不十分熟悉。这是我到目前为止获得的Python代码:
I want to do some performance testing on one of our web servers, to see how the server handles a lot of persistent connections. Unfortunately, I'm not terribly familiar with HTTP and web testing. Here's the Python code I've got for this so far:
import http.client
import argparse
import threading
def make_http_connection():
conn = http.client.HTTPConnection(options.server, timeout=30)
conn.connect()
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("num", type=int, help="Number of connections to make (integer)")
parser.add_argument("server", type=str, help="Server and port to connect to. Do not prepend \'http://\' for this")
options = parser.parse_args()
for n in range(options.num):
connThread = threading.Thread(target = make_http_connection, args = ())
connThread.daemon = True
connThread.start()
while True:
try:
pass
except KeyboardInterrupt:
break
我的主要问题是:我如何保持这些连接存活?我设置了一个很长的超时,但这是一个非常粗略的方法我甚至不确定它会影响连接。只需要每隔一段时间请一个或两个字节吗?
My main question is this: How do I keep these connections alive? I've set a long timeout, but that's a very crude method and I'm not even sure it affects the connection. Would simply requesting a byte or two every once in a while do it?
(另外,在一个不相关的注释中,有一个更好的程序等待键盘中断比丑陋的而True:
阻止我的代码结尾?)
(Also, on an unrelated note, is there a better procedure for waiting for a keyboard interrupt than the ugly while True:
block at the end of my code?)
推荐答案
urllib.request
不支持持久连接。代码中有'Connection:close'
硬编码。但是 http.client
部分支持持久连接(包括遗留的http / 1.0 keep-alive
)。因此问题标题可能会产生误导。
urllib.request
doesn't support persistent connections. There is 'Connection: close'
hardcoded in the code. But http.client
partially supports persistent connections (including legacy http/1.0 keep-alive
). So the question title might be misleading.
我想在一个上做一些性能测试我们的Web服务器,看看服务器如何处理大量持久连接。不幸的是,我对HTTP和Web测试并不十分熟悉。
I want to do some performance testing on one of our web servers, to see how the server handles a lot of persistent connections. Unfortunately, I'm not terribly familiar with HTTP and web testing.
您可以使用现有的http测试工具,例如 slowloris ,httperf而不是自己写一个。
You could use an existing http testing tools such as slowloris, httperf instead of writing one yourself.
如何保持这些连接存活?
How do I keep these connections alive?
要关闭http / 1.1连接,客户端应显式指定 Connection:close
header否则连接被认为是持久的服务器(虽然它可能随时关闭它并且 http.client
将不知道它直到它尝试读/写连接为止。
To close http/1.1 connection a client should explicitly specify Connection: close
header otherwise the connection is considered persistent by the server (though it may close it at any moment and http.client
won't know about it until it tries to read/write to the connection).
conn.connect()
几乎立即返回,你的线程结束。要强制每个线程维持与服务器的http连接,您可以:
conn.connect()
returns almost immediately and your thread ends. To force each thread to maintain an http connection to the server you could:
import time
def make_http_connection(*args, **kwargs):
while True: # make new http connections
h = http.client.HTTPConnection(*args, **kwargs)
while True: # make multiple requests using a single connection
try:
h.request('GET', '/') # send request; make conn. on the first run
response = h.getresponse()
while True: # read response slooowly
b = response.read(1) # read 1 byte
if not b:
break
time.sleep(60) # wait a minute before reading next byte
#note: the whole minute might pass before we notice that
# the server has closed the connection already
except Exception:
break # make new connection on any error
注意:如果服务器返回'连接:关闭'
则每个连接只有一个请求。
Note: if the server returns 'Connection: close'
then there is a single request per connection.
(另外,在一个不相关的注释中,有一个更好的程序等待键盘中断而不是丑陋而True:阻塞在我的代码末尾?)
(Also, on an unrelated note, is there a better procedure for waiting for a keyboard interrupt than the ugly while True: block at the end of my code?)
要等到所有线程f发生inish或 KeyboardInterrupt
你可以:
To wait until all threads finish or KeyboardInterrupt
happens you could:
while threads:
try:
for t in threads[:]: # enumerate threads
t.join(.1) # timeout 0.1 seconds
if not t.is_alive():
threads.remove(t)
except KeyboardInterrupt:
break
或类似的东西:
while threading.active_count() > 1:
try:
main_thread = threading.current_thread()
for t in threading.enumerate(): # enumerate all alive threads
if t is not main_thread:
t.join(.1)
except KeyboardInterrupt:
break
稍后可能由于各种原因而无法工作,例如,如果存在虚拟线程,例如线程在C扩展中启动而不使用 threading
模块。
The later might not work for various reasons e.g., if there are dummy threads such as threads that started in C extensions without using threading
module.
concurrent.futures.ThreadPoolExecutor 提供更高的抽象级别高于线程
模块,它可以隐藏一些复杂性。
concurrent.futures.ThreadPoolExecutor provides a higher abstraction level than threading
module and it can hide some complexity.
您可以打开多个连接而不是每个连接模型的线程同时在一个线程中,例如,直接使用 requests.async
或 gevent
。
Instead of thread per connection model you could open multiple connections concurrently in a single thread e.g., using requests.async
or gevent
directly.
这篇关于urllib.request连接到HTTP服务器的持久性的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!