带有多线程的python-requests [英] python-requests with multithreading

问题描述

我正在创建一个HTTP客户端，该客户端每秒可以生成数百个连接，并在每个连接上最多发送10个请求.我正在使用线程，因此可以实现并发. 这是我的代码:

I am working on creating a HTTP client which can generate hundreds of connections each second and send up to 10 requests on each of those connections. I am using threading so concurrency can be achieved. Here is my code:

def generate_req(reqSession):
    requestCounter = 0
    while requestCounter < requestRate:
        try:
            response1 = reqSession.get('http://20.20.1.2/tempurl.html')
            if response1.status_code == 200:
                client_notify('r')
        except(exceptions.ConnectionError, exceptions.HTTPError, exceptions.Timeout) as Err:
            client_notify('F')
            break
        requestCounter += 1

def main():
    for q in range(connectionPerSec):
        s1 = requests.session()
        t1 = threading.Thread(target=generate_req, args=(s1,))
        t1.start()

问题:

1. 在requestRate = 1的情况下，扩展速度没有超过200个/秒.我在同一台客户端计算机上针对服务器运行了其他可用的HTTP客户端，测试运行正常并且可以扩展.

1. It is not scaling above 200 connections/sec with requestRate = 1. I ran other available HTTP clients on the same client machine and against the server, test runs fine and it is able to scale.

当requestRate = 10时，连接/秒降低到30. 原因:无法每秒创建目标数量的线程.

When requestRate = 10, connections/sec drops to 30. Reason: Not able to create targeted number of threads every second.

对于问题2，客户端计算机无法创建足够的请求会话并启动新线程.一旦requestRate设置为大于1，事情就会开始崩溃. 我怀疑这与请求使用的HTTP连接池有关.

For issue #2, client machine is not able to create enough request sessions and start new threads. As soon as requestRate is set to more than 1, things start to fall apart. I am suspecting it has something to do with HTTP connection pooling which requests uses.

请在这里提出我在做什么错

Please suggest what am I doing wrong here.

推荐答案

我无法使事情崩溃，但是以下代码具有一些新功能:

I wasn't able to get things to fall apart, however the following code has some new features:

1)扩展日志记录，包括特定的每线程信息

1) extended logging, including specific per-thread information

2)最后所有join()线程，以确保父进程不会使它们挂起

2) all threads join()ed at the end to make sure the parent process doesntt leave them hanging

3)多线程print倾向于交织消息，这可能很麻烦.此版本使用yield，因此以后的版本可以接受消息并清晰打印.

3) multithreaded print tends to interleave the messages, which can be unwieldy. This version uses yield so a future version can accept the messages and print them clearly.

import exceptions, requests, threading, time

requestRate = 1
connectionPerSec = 2


def client_notify(msg):
    return time.time(), threading.current_thread().name, msg

def generate_req(reqSession):
    requestCounter = 0
    while requestCounter < requestRate:
        try:
            response1 = reqSession.get('http://127.0.0.1/')
            if response1.status_code == 200:
                print client_notify('r')
        except (exceptions.ConnectionError, exceptions.HTTPError, exceptions.Timeout):
            print client_notify('F')
            break
        requestCounter += 1

def main():
    for cnum in range(connectionPerSec):
        s1 = requests.session()
        th = threading.Thread(
            target=generate_req, args=(s1,),
            name='thread-{:03d}'.format(cnum),
        )
        th.start()

    for th in threading.enumerate():
        if th != threading.current_thread():
            th.join()


if __name__=='__main__':
    main()

输出

(1407275951.954147, 'thread-000', 'r')
(1407275951.95479, 'thread-001', 'r')

这篇关于带有多线程的python-requests的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！