限制自己避免重载外部API的速率 [英] Rate limiting yourself from overloading external API's
问题描述
我发现了很多信息和脚本示例，这些示例显示了如何对API用户进行速率限制，但是当施加这些限制时，我找不到任何如何对您自己的API请求进行速率限制的示例.
I found a lot of information and script examples around that showed how to rate limit the users of an API but I wasn't able to find any examples of how to rate limit your own requests of an API when these limits are imposed.
我一直使用sleep或usleep命令之类的代码来限制脚本的速度，但是这感觉是一种低效的处理方式，尤其是当API端点具有很高的速率限制并锤击API直到您点击时限制也是无效的.
I've always rate limited my scripts with code such as sleep or usleep commands but it feels like an inefficient way of doing things, especially when the API endpoints have pretty high rate limits and hammering API's until you hit the limits is also inefficient.
例如，Google的API限制会因您所使用的API而异，并且可以增加/减少，在这种情况下，硬编码到代码中的固定速率限制似乎就像原始猜测工作一样！
For example, Google's API limits vary based on the API you are using and can increase/decrease, in this case a fixed rate limit hard coded into the code just seems like primitive guess work!
我错过了一些显而易见的事情吗?还是这不像我期望的那么普遍?
Have I missed something pretty obvious? Or is this just not as common as I expect it to be?
推荐答案
好吧，为咯咯地笑，我抛出了一个限制器类，该类将允许您指定每秒，分钟和小时的限制.我无法抗拒使用循环队列的充分理由！
Okay, for giggles I've thrown together a limiter class that will allow you to specify the limit per second, minute and hour. I can't resist having a good reason to use a circular queue!
如果您有多个进程进行使用，无论是否同时进行，您都必须设计一种方法来自行存储和/或共享使用历史记录.
If you have multiple processes doing the consumption, whether simultaneous or not, you'll have to devise a way to store and/or share the usage history on your own.
// LIMITER.PHP
class Limiter
{
private $queue = array();
private $size;
private $next;
private $perSecond;
private $perMinute;
private $perHour;
// Set any constructor parameter to non-zero to allow adherence to the
// limit represented. The largest value present will be the size of a
// circular queue used to track usage.
// -------------------------------------------------------------------
function __construct($perSecond=0,$perMinute=0,$perHour=0)
{
$this->size = max($perSecond,$perMinute,$perHour);
$this->next = 0;
$this->perSecond = $perSecond;
$this->perMinute = $perMinute;
$this->perHour = $perHour;
for($i=0; $i < $this->size; $i++)
$this->queue[$i] = 0;
}
// See if a use would violate any of the limits specified. We return true
// if a limit has been hit.
// ----------------------------------------------------------------------
public function limitHit($verbose=0)
{
$inSecond = 0;
$inMinute = 0;
$inHour = 0;
$doneSecond = 0;
$doneMinute = 0;
$doneHour = 0;
$now = microtime(true);
if ( $verbose )
echo "Checking if limitHit at $now<br>\n";
for ($offset=1; $offset <= $this->size; $offset++)
{
$spot = $this->next - $offset;
if ( $spot < 0 )
$spot = $this->size - $offset + $this->next;
if ( $verbose )
echo "... next $this->next size $this->size offset $offset spot $spot utime " . $this->queue[$spot] . "<br>\n";
// Count and track within second
// -----------------------------
if ( $this->perSecond && !$doneSecond && $this->queue[$spot] >= microtime(true) - 1.0 )
$inSecond++;
else
$doneSecond = 1;
// Count and track within minute
// -----------------------------
if ( $this->perMinute && !$doneMinute && $this->queue[$spot] >= microtime(true) - 60.0 )
$inMinute++;
else
$doneMinute = 1;
// Count and track within hour
// ---------------------------
if ( $this->perHour && !$doneHour && $this->queue[$spot] >= microtime(true) - 3600.0 )
$inHour++;
else
$doneHour = 1;
if ( $doneSecond && $doneMinute && $doneHour )
break;
}
if ( $verbose )
echo "... inSecond $inSecond inMinute $inMinute inHour $inHour<br>\n";
if ( $inSecond && $inSecond >= $this->perSecond )
{
if ( $verbose )
echo "... limit perSecond hit<br>\n";
return TRUE;
}
if ( $inMinute && $inMinute >= $this->perMinute )
{
if ( $verbose )
echo "... limit perMinute hit<br>\n";
return TRUE;
}
if ( $inHour && $inHour >= $this->perHour )
{
if ( $verbose )
echo "... limit perHour hit<br>\n";
return TRUE;
}
return FALSE;
}
// When an API is called the using program should voluntarily track usage
// via the use function.
// ----------------------------------------------------------------------
public function usage()
{
$this->queue[$this->next++] = microtime(true);
if ( $this->next >= $this->size )
$this->next = 0;
}
}
// ##############################
// ### Test the limiter class ###
// ##############################
$psec = 2;
$pmin = 4;
$phr = 0;
echo "Creating limiter with limits of $psec/sec and $pmin/min and $phr/hr<br><br>\n";
$monitorA = new Limiter($psec,$pmin,$phr);
for ($i=0; $i<15; $i++)
{
if ( !$monitorA->limitHit(1) )
{
echo "<br>\n";
echo "API call A here (utime " . microtime(true) . ")<br>\n";
echo "Voluntarily registering usage<br>\n";
$monitorA->usage();
usleep(250000);
}
else
{
echo "<br>\n";
usleep(500000);
}
}
为了演示它的实际效果，我在限制检查功能中加入了一些详细模式"语句.这是一些示例输出.
In order to demonstrate it in action I've put in some "verbose mode" statements in the limit checking function. Here is some sample output.
Creating limiter with limits of 2/sec and 4/min and 0/hr
Checking if limitHit at 1436267440.9957
... next 0 size 4 offset 1 spot 3 utime 0
... inSecond 0 inMinute 0 inHour 0
API call A here (utime 1436267440.9957)
Voluntarily registering usage
Checking if limitHit at 1436267441.2497
... next 1 size 4 offset 1 spot 0 utime 1436267440.9957
... next 1 size 4 offset 2 spot 3 utime 0
... inSecond 1 inMinute 1 inHour 0
API call A here (utime 1436267441.2497)
Voluntarily registering usage
Checking if limitHit at 1436267441.5007
... next 2 size 4 offset 1 spot 1 utime 1436267441.2497
... next 2 size 4 offset 2 spot 0 utime 1436267440.9957
... next 2 size 4 offset 3 spot 3 utime 0
... inSecond 2 inMinute 2 inHour 0
... limit perSecond hit
Checking if limitHit at 1436267442.0007
... next 2 size 4 offset 1 spot 1 utime 1436267441.2497
... next 2 size 4 offset 2 spot 0 utime 1436267440.9957
... next 2 size 4 offset 3 spot 3 utime 0
... inSecond 1 inMinute 2 inHour 0
API call A here (utime 1436267442.0007)
Voluntarily registering usage
Checking if limitHit at 1436267442.2507
... next 3 size 4 offset 1 spot 2 utime 1436267442.0007
... next 3 size 4 offset 2 spot 1 utime 1436267441.2497
... next 3 size 4 offset 3 spot 0 utime 1436267440.9957
... next 3 size 4 offset 4 spot 3 utime 0
... inSecond 1 inMinute 3 inHour 0
API call A here (utime 1436267442.2507)
Voluntarily registering usage
Checking if limitHit at 1436267442.5007
... next 0 size 4 offset 1 spot 3 utime 1436267442.2507
... next 0 size 4 offset 2 spot 2 utime 1436267442.0007
... next 0 size 4 offset 3 spot 1 utime 1436267441.2497
... next 0 size 4 offset 4 spot 0 utime 1436267440.9957
... inSecond 2 inMinute 4 inHour 0
... limit perSecond hit
Checking if limitHit at 1436267443.0007
... next 0 size 4 offset 1 spot 3 utime 1436267442.2507
... next 0 size 4 offset 2 spot 2 utime 1436267442.0007
... next 0 size 4 offset 3 spot 1 utime 1436267441.2497
... next 0 size 4 offset 4 spot 0 utime 1436267440.9957
... inSecond 2 inMinute 4 inHour 0
... limit perSecond hit
Checking if limitHit at 1436267443.5027
... next 0 size 4 offset 1 spot 3 utime 1436267442.2507
... next 0 size 4 offset 2 spot 2 utime 1436267442.0007
... next 0 size 4 offset 3 spot 1 utime 1436267441.2497
... next 0 size 4 offset 4 spot 0 utime 1436267440.9957
... inSecond 0 inMinute 4 inHour 0
... limit perMinute hit
Checking if limitHit at 1436267444.0027
... next 0 size 4 offset 1 spot 3 utime 1436267442.2507
... next 0 size 4 offset 2 spot 2 utime 1436267442.0007
... next 0 size 4 offset 3 spot 1 utime 1436267441.2497
... next 0 size 4 offset 4 spot 0 utime 1436267440.9957
... inSecond 0 inMinute 4 inHour 0
... limit perMinute hit
这篇关于限制自己避免重载外部API的速率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
