如何在 Python 3 的多线程程序中限制 API 调用? [英] How can I limit API calls in multithreaded program in Python 3?

问题描述

经过大量研究，我不确定最佳实践是什么，我的以下想法是否合适?

After much research, I'm not sure what the best practice is, is my following idea decent?

我想访问一个 API，该 API 将我可以进行的总调用次数限制为每分钟 50 次.

I want to access an API that limits the total number of calls I can make to 50 per minute.

我的程序有多个独立运行的线程.

My program has multiple threads operating on their own.

如何限制我的程序保持在阈值以下?

How can I limit my program to stay below the threshold?

我的想法是创建一个队列，每 X 秒添加一个东西，其中 X = thread_count/allowed_calls*60.然后需要一个单独的线程来处理这些请求.(还有一个单独的线程，用于定期添加)

My idea is to create a queue, and add one thing to it every X seconds where X = thread_count/allowed_calls*60. Then a separate thread would be needed for handling these requests. (And a separate thread for adding at a regular interval)

对于这种情况，最佳做法是什么?有没有办法在不需要为每个小功能完全独立的线程的情况下实现这一点?

What would the best practice be for something like this? Is there a way to achieve this without needing completely separate threads for each little functionality?

推荐答案

为什么不创建一个使用内部变量来控制调用次数和第一次调用的类?

Why don't you create a class that uses an internal variable to control the number of calls and the first call on that second?

从https://github.com/lucjon/Py-StackExchange/blob/master/stackexchange/web.py

基本上，它会检查您是否有超过所需的通话量，如果是这种情况，它会停止.如果您使用的是多线程(如 Pool)，请将函数请求作为要执行的函数传递.

Basically, it check if you have more than the amount of calls you need and stop if this is the case. If you are using multithread (like Pool), pass the function request as the funcion to be executed.

class WebRequestManager(object):
    # When we last made a request
    window = datetime.datetime.now()
    # Number of requests since last throttle window
    num_requests = 0

    def request(self, url, params):
        now = datetime.datetime.now()

        # Before we do the actual request, are we going to be throttled?
        def halt(wait_time):
            if self.throttle_stop:
                raise TooManyRequestsError()
            else:
                # Wait the required time, plus a bit of extra padding time.
                time.sleep(wait_time + 0.1)

        if (now - WebRequestManager.window).seconds >= 1:
            WebRequestManager.window = now
            WebRequestManager.num_requests = 0

        WebRequestManager.num_requests += 1
        if WebRequestManager.num_requests > 30:
            halt(5 - (WebRequestManager.window - now).seconds)

        request = urllib.request.Request(url)
        ...

这篇关于如何在 Python 3 的多线程程序中限制 API 调用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！