用户速率限制在几个请求后超出 [英] User rate limit exceeded after a few requests

问题描述

我通过 pydrive 使用Google Drive API在两个Google云端硬盘帐户之间移动文件。我一直在测试16个文件的文件夹。我的代码总是在第六个文件中产生错误

$ b

超出用户访问量限制>

我知道请求数量有限制（10 / s或1000 / 100s），但我尝试过指数回退，Google Drive API建议您处理此错误。即使在248s之后，它仍然会引发同样的错误。

这里是我正在做的一个例子

  def MoveToFolder（self，files，folder_id，drive）：
 total_files = len（文件）
 for范围内（total_files）：
 success = False 
n = 0 
，但不成功：
 try：
 drive.auth.service.files（）。copy（fileId = files [cont] ['id']，
 body = {parents：[{kind：drive＃fileLink，id：folder_id}]}）。execute（）
 time.sleep（random.randint（0,1000）/ 1000） 
成功=真
除外：
等待=（2 ** n）+（random.randint（0,1000）/ 1000）
 time.sleep（等待）
 success = False 
n + = 1 
  

我尝试使用批处理请求复制这些文件，但它会为10个文件引发相同的错误。

  def MoveToFolderBatch（self，files，folder_id，drive）：
 cont = 0 
 batch = drive .auth.service.new_batch_http_request（）
用于文件中的文件：
 cont + = 1 
 batch.add（drive.auth.service.files（）。copy（fileId = file ['id ']，
 body = {parents：[
 {kind：drive＃fileLink，id：folder_id}]}）
 batch.execute（）
  

有人有任何提示吗？

编辑：
据Google支持：

用户速率限制超出错误，与控制台中设置的每用户速率限制完全不相关。相反，它来自Drive API依赖的内部Google系统，并且最有可能在单个帐户拥有域中的所有文件时发生。
我们不建议一个帐户拥有所有文件，而是让域中的个人用户拥有这些文件。为了传输文件，你可以检查这个链接。 href =https://stackoverflow.com/questions/18529524/403-rate-limit-after-only-1-insert-per-second> 403每秒只有1次插入后的速率限制和 403插入速率限制有时会成功

关键点是： -

• 退避但不实现指数退避！ STRONG>。这将简单地杀死您的应用程序吞吐量，而不是您需要主动调整您的请求以避免发生304。在我的测试中，我发现最大可持续吞吐量大约是每1.5秒1次交易。




• 批次使问题变得更糟，因为批次在304被提起。 IE浏览器。一批10个被解释为10个快速交易，而不是1个。

试试这个算法

  delay = 0 //开始时没有回退
 while（haveFilesInUploadQueue）{
 sleep（delay）// backoff 
上传（文件）//尝试上传
 if（403 Rate Limit）{//如果拒绝了速率限制
 delay + = 2s //添加2秒延迟以保证下次尝试OK 
 else else {
 removeFileFromQueue（file）//如果没有被拒绝，标记文件完成
 if（delay> 0）{//如果我们当前正在关闭
延迟 -  = 0.2s //减少退避延迟
} 
} 
} 
 //你可以玩2s和0.2s来优化吞吐量。关键是要尽你所能避免403 
  

有一点需要注意的是，是（是？）Drive的一个错误，有时上传被403拒绝，但是，尽管发送了403，但Drive继续并创建文件。症状将被复制文件。所以为了更加安全，在403之后你应该以某种方式检查文件是否真的存在。最简单的方法是使用预先分配的ID，或者将您自己的不透明ID添加到属性中。

I am using Google Drive API through pydrive to move files between two google drive accounts. I have been testing with a folder with 16 files. My code always raises an error in the sixth file

"User rate limit exceeded">

I know that there is a limit for the number of request (10/s or 1000/100s), but I have tried the exponential backoff suggested by the Google Drive API to handle this error. Even after 248s it still raises the same error.

Here an example what I am doing

def MoveToFolder(self,files,folder_id,drive):
    total_files = len(files)
    for cont in range(total_files):
        success = False
        n=0
        while not success:
            try:
                drive.auth.service.files().copy(fileId=files[cont]['id'],
                                                body={"parents": [{"kind": "drive#fileLink", "id": folder_id}]}).execute()
                time.sleep(random.randint(0,1000)/1000)
                success = True
            except:
                wait = (2**n) + (random.randint(0,1000)/1000)
                time.sleep(wait)
                success = False
                n += 1

I tried to use "Batching request" to copy the files, but it raises the same errors for 10 files.

def MoveToFolderBatch(self,files,folder_id,drive):
    cont=0
    batch = drive.auth.service.new_batch_http_request()
    for file in files:
        cont+=1
        batch.add(drive.auth.service.files().copy(fileId=file['id'],
                                                 body={"parents": [
                                                     {"kind": "drive#fileLink", "id": folder_id}]}))
    batch.execute()

Does anyone have any tips?

EDIT: According to google support:

Regarding your User rate limit exceeded error, is not at all related to the per-user rate limit set in the console. Instead it is coming from internal Google systems that the Drive API relies on and are most likely to occur when a single account owns all the files within a domain. We don't recommend that a single account owns all the files, and instead have individual users in the domain own the files. For transferring files, you can check this link. Also, please check this link on the recommendations to avoid the error.

解决方案

See 403 rate limit after only 1 insert per second and 403 rate limit on insert sometimes succeeds

The key points are:-

• backoff but do not implement exponential backoff!. This will simply kill your application throughput

• instead you need to proactively throttle your requests to avoid the 304's from occurring. In my testing I've found that the maximum sustainable throughput is about 1 transaction every 1.5 seconds.

• batching makes the problem worse because the batch is unpacked before the 304 is raised. Ie. a batch of 10 is interpreted as 10 rapid transactions, not 1.

Try this algorithm

delay=0                              // start with no backoff
while (haveFilesInUploadQueue) {
   sleep(delay)                      // backoff 
   upload(file)                      // try the upload
   if (403 Rate Limit) {             // if rejected with a rate limit
      delay += 2s                    // add 2s to delay to guarantee the next attempt will be OK
   } else {
      removeFileFromQueue(file)      // if not rejected, mark file as done
      if (delay > 0) {               // if we are currently backing off
         delay -= 0.2s               // reduce the backoff delay
      }
   }
}
// You can play with the 2s and 0.2s to optimise throughput. The key is to do all you can to avoid the 403's 

One thing to be aware of is that there was (is?) a bug with Drive that sometimes an upload gets rejected with a 403, but, despite sending the 403, Drive goes ahead and creates the file. The symptom will be duplicated files. So to be extra safe, after a 403 you should somehow check if the file is actually there. Easiest way to do this is use pre-allocated ID's, or to add your own opaque ID to a property.

这篇关于用户速率限制在几个请求后超出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！