如何限制在Python 3中多线程程序中调用API？ [英] How can I limit API calls in multithreaded program in Python 3?

问题描述

经过反复研究，我不知道最好的做法是什么，是我的想法如下体面？

After much research, I'm not sure what the best practice is, is my following idea decent?

我想要访问限制呼叫的总数我可以使50每分钟的API。

I want to access an API that limits the total number of calls I can make to 50 per minute.

我的程序有多个线程对自己的工作。

My program has multiple threads operating on their own.

如何限制我的程序保持低于阈值？

How can I limit my program to stay below the threshold?

我的想法是创建一个队列，并添加一件事，它每隔X秒，其中X = THREAD_COUNT / allowed_calls * 60。然后将需要处理这些请求一个单独的线程。 （而对于以规则的间隔增加一个单独的线程）

My idea is to create a queue, and add one thing to it every X seconds where X = thread_count/allowed_calls*60. Then a separate thread would be needed for handling these requests. (And a separate thread for adding at a regular interval)

将最好的做法是什么这样的事情？有没有办法实现这个无需完全独立的线程每个小功能呢？

What would the best practice be for something like this? Is there a way to achieve this without needing completely separate threads for each little functionality?

推荐答案

你为什么不创建使用一个内部变量来控制呼叫的数量和上第二个在第一次调用类？

Why don't you create a class that uses an internal variable to control the number of calls and the first call on that second?

此撕开code从
https://github.com/lucjon/Py-StackExchange/blob /master/stackexchange/web.py

基本上，它检查是否有比你需要和停止，如果是这样的话通话量较多。
如果您正在使用多线程（如游泳池），通过功能要求为将要执行的funcion。

Basically, it check if you have more than the amount of calls you need and stop if this is the case. If you are using multithread (like Pool), pass the function request as the funcion to be executed.

class WebRequestManager(object):
    # When we last made a request
    window = datetime.datetime.now()
    # Number of requests since last throttle window
    num_requests = 0

    def request(self, url, params):
        now = datetime.datetime.now()

        # Before we do the actual request, are we going to be throttled?
        def halt(wait_time):
            if self.throttle_stop:
                raise TooManyRequestsError()
            else:
                # Wait the required time, plus a bit of extra padding time.
                time.sleep(wait_time + 0.1)

        if (now - WebRequestManager.window).seconds >= 1:
            WebRequestManager.window = now
            WebRequestManager.num_requests = 0

        WebRequestManager.num_requests += 1
        if WebRequestManager.num_requests > 30:
            halt(5 - (WebRequestManager.window - now).seconds)

        request = urllib.request.Request(url)
        ...

这篇关于如何限制在Python 3中多线程程序中调用API？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！