插入403的速率限制有时会成功 [英] 403 rate limit on insert sometimes succeeds
本文介绍了插入403的速率限制有时会成功的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
它听起来像API端点服务器和Drive存储服务器之间存在异步连接，是造成非确定性的结果，特别是在大量写入。
因为这意味着我不能依靠403速率限制来限制我的插入，我需要知道什么是安全插入率，以免触发这些计时错误。
运行下面的代码，给出...
小结...
文件插入尝试次数（a）= 100
限制错误率（b）= 31
预期文件数（ab）= 69
实际文件数量= 73
code ...
package com.cnw.test.servlets;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.googleapis.json.GoogleJsonError;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson.JacksonFactory;
导入com.google.api.services.drive.Drive;
import com.google.api.services.drive.model.ChildList;
导入com.google.api.services.drive.model.File;
import com.google.api.services.drive.model.File.Labels;
import com.google.api.services.drive.model.ParentReference;
导入couk.cleverthinking.cnw.oauth.CredentialMediatorB;
导入couk.cleverthinking.cnw.oauth.CredentialMediatorB.InvalidClientSecretsException;
@SuppressWarnings（serial）
/ **
*
* AppEngine servlet证明Drive IS执行插入操作，尽管抛出了403速率限制异常。
*
*它所做的只是创建一个文件夹，然后循环创建x文件。计数任何403个速率限制例外。
*最后，比较文件的预期数量（尝试 - 403）与实际值。
*在100个文件的运行中，我始终可以看到比预期更多的1到3个文件，即。尽管抛出了403的费率限制，但
* Drive *有时会创建该文件。
*
*要运行此操作，您需要...
* 1）在
*之上输入一个APPNAME）输入一个超过
* 3）为该用户
*
*（2）和（3）提供一个有效的存储凭证可以由手动构建的Credential
*
替换*您的测试必须生成费率限制错误，所以如果连接速度非常慢，则可能需要并行运行2或3。
*我在中速连接上运行测试，在插入30个左右后，我看到了403个速率限制。
*一直创建100个文件暴露了问题。
*
* /
public class Hack extends HttpServlet {
private final String APPNAME =MyApp; //输入您的应用名称
private final String GOOGLE_USER_ID_TO_FETCH_CREDENTIAL =11222222222222222222222; //输入你的GOOGLE用户名
@Override $ b $ public void doGet（HttpServletRequest request，HttpServletResponse response）throws IOException {
/ *
*设置计数器
* /
//我将它作为一个servlet来运行，所以我从请求URL获取文件数量
int numFiles = Integer.parseInt（request.getParameter（numfiles））;
int fileCount = 0;
int ratelimitCount = 0;
/ *
*加载凭证
* /
CredentialMediatorB cmb = null;
尝试{
cmb = new CredentialMediatorB（request）;
} catch（InvalidClientSecretsException e）{
e.printStackTrace（）;
}
//这将获取存储的凭证，您可以选择手动构建一个
凭证凭证= cmb.getStoredCredential（GOOGLE_USER_ID_TO_FETCH_CREDENTIAL）;
$ b $ * b $ b *使用凭证创建驱动器服务
* /
Drive driveService = new Drive.Builder（new NetHttpTransport（），new JacksonFactory（） ，凭证）.setApplicationName（APPNAME）.build（）;
$ b $ * b $ b *使一个父文件夹更易于计算文件并在测试后删除它们
* /
文件folderParent = new File（） ;
folderParent.setTitle（403parentfolder-+ numFiles）;
folderParent.setMimeType（application / vnd.google-apps.folder）;
folderParent.setParents（Arrays.asList（new ParentReference（）。setId（root）））;
folderParent.setLabels（new Labels（）。setHidden（false））;
driveService.files（）.list（）。execute（）;
folderParent = driveService.files（）。insert（folderParent）.execute（）;
System.out.println（使用id =+ folderParent.getId（））创建的文件夹;
$ b $ * /
*将父文件夹标识存储在父数组中供每个子文件使用
* /
List< ParentReference> parents = new ArrayList< ParentReference>（）;
parents.add（new ParentReference（）。setId（folderParent.getId（）））; （fileCount = 0; fileCount< numFiles; fileCount ++）{
/ * $（每个文件
* /
）
/ *
* b $ b *为插入文件创建一个File对象
* /
File file = new File（）;
file.setTitle（testfile-+（fileCount + 1））;
file.setParents（父母）;
file.setDescription（description）;
file.setMimeType（text / html）;
尝试{
System.out.println（make file+ fileCount +of+ numFiles）;
//调用驱动服务插入执行方法
driveService.files（）。insert（file）.setConvert（false）.execute（）;
} catch（GoogleJsonResponseException e）{
GoogleJsonError error = e.getDetails（）;
//查找汇率错误并对它们进行计数。通常人们会在这里展示退后，但这是为了证明尽管
// 403，文件DID得到了创建
if（error.getCode（）== 403&& error.getMessage （）.toLowerCase（）。contains（rate limit））{
System.out.println（文件上的速率限制异常+fileCount +的+ numFiles）;
//增加一个速率限制错误计数
ratelimitCount ++;
} else {
//以防万一抛出不同的异常
System.out.println（[DbSA465]错误信息：+ error.getCode（）++ error.getMessage（））;
}
}
}
/ *
*全部完成。让文件夹的孩子看看实际创建了多少文件
* /
ChildList children = driveService.children（）.list（folderParent.getId（））。execute（）;
$ b $ * / $
*和赢家是...
* /
System.out.println（\\\
Summary ...）;
System.out.println（File insert attempts（a）=+ numFiles）;
System.out.println（rate limit errors（b）=+ ratelimitCount）;
System.out.println（预期的文件数量（a-b）=+（numFiles - ratelimitCount））;
System.out.println（实际的文件数量=+ children.getItems（）。size（）+注意：一页内有100个孩子的限制，所以如果你期望更多比100，需要遵循nextPageToken）;
}
}
解决方案
我假设你正在尝试执行并行下载...
这可能不是您正在寻找的答案，但是这个是我在与google drive api的互动中经历的。我使用C＃，所以它有点不同，但也许它会有所帮助。
我必须设置特定数量的线程才能同时运行。如果我让我的程序一次运行100个条目作为单独的线程，我也遇到了速率限制错误。
我根本不知道，但在我的C＃程序中，我运行了3个线程（可由用户定义，3为默认值）
opts = new ParallelOptions {MaxDegreeOfParallelism = 3};
var checkforfinished =
Parallel.ForEach（lstBackupUsers.Items.Cast< ListViewItem>（），opts，name => {
{//我的逻辑代码在这里}
我做了一个快速搜索，发现Java 8（不确定这是否是您使用的）支持Parallel（）。 forEach（），也许这对你有帮助。我找到的资源位于： http://radar.oreilly.com/2015/02/java-8-streams-api-and-parallelism.html
希望这会有所帮助，轮流尝试帮助别人，因为人们帮助了我！
I insert 100 files in a loop. For this test I have DISABLED backoff and retry, so if an insert fails with a 403, I ignore it and proceed with the next file. Out of 100 files, I get 63 403 rate limit exceptions.
However, on checking Drive, of those 63 failures, 3 actually succeeded, ie. the file was created on drive. Had I done the usual backoff and retry, I would have ended up with duplicated inserts. This confirms the behaviour I was seeing with backoff-retry enabled, ie. from my 100 file test, I am consistently seeing 3-4 duplicate insertions.
It smells like there is an asynchronous connection between the API endpoint server and the Drive storage servers which is causing non-deterministic results, especially on high volume writes.
Since this means I can't rely on "403 rate limit" to throttle my inserts, I need to know what is a safe insert rate so as not to trigger these timing bugs.
Running the code below, gives ...
Summary...
File insert attempts (a) = 100
rate limit errors (b) = 31
expected number of files (a-b) = 69
Actual number of files = 73
code...
package com.cnw.test.servlets;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.googleapis.json.GoogleJsonError;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson.JacksonFactory;
import com.google.api.services.drive.Drive;
import com.google.api.services.drive.model.ChildList;
import com.google.api.services.drive.model.File;
import com.google.api.services.drive.model.File.Labels;
import com.google.api.services.drive.model.ParentReference;
import couk.cleverthinking.cnw.oauth.CredentialMediatorB;
import couk.cleverthinking.cnw.oauth.CredentialMediatorB.InvalidClientSecretsException;
@SuppressWarnings("serial")
/**
*
* AppEngine servlet to demonstrate that Drive IS performing an insert despite throwing a 403 rate limit exception.
*
* All it does is create a folder, then loop to create x files. Any 403 rate limit exceptions are counted.
* At the end, compare the expected number of file (attempted - 403) vs. the actual.
* In a run of 100 files, I consistently see between 1 and 3 more files than expected, ie. despite throwing a 403 rate limit,
* Drive *sometimes* creates the file anyway.
*
* To run this, you will need to ...
* 1) enter an APPNAME above
* 2) enter a google user id above
* 3) Have a valid stored credential for that user
*
* (2) and (3) can be replaced by a manually constructed Credential
*
* Your test must generate rate limit errors, so if you have a very slow connection, you might need to run 2 or 3 in parallel.
* I run the test on a medium speed connection and I see 403 rate limits after 30 or so inserts.
* Creating 100 files consistently exposes the problem.
*
*/
public class Hack extends HttpServlet {
private final String APPNAME = "MyApp"; // ENTER YOUR APP NAME
private final String GOOGLE_USER_ID_TO_FETCH_CREDENTIAL = "11222222222222222222222"; //ENTER YOUR GOOGLE USER ID
@Override
public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
/*
* set up the counters
*/
// I run this as a servlet, so I get the number of files from the request URL
int numFiles = Integer.parseInt(request.getParameter("numfiles"));
int fileCount = 0;
int ratelimitCount = 0;
/*
* Load the Credential
*/
CredentialMediatorB cmb = null;
try {
cmb = new CredentialMediatorB(request);
} catch (InvalidClientSecretsException e) {
e.printStackTrace();
}
// this fetches a stored credential, you might choose to construct one manually
Credential credential = cmb.getStoredCredential(GOOGLE_USER_ID_TO_FETCH_CREDENTIAL);
/*
* Use the credential to create a drive service
*/
Drive driveService = new Drive.Builder(new NetHttpTransport(), new JacksonFactory(), credential).setApplicationName(APPNAME).build();
/*
* make a parent folder to make it easier to count the files and delete them after the test
*/
File folderParent = new File();
folderParent.setTitle("403parentfolder-" + numFiles);
folderParent.setMimeType("application/vnd.google-apps.folder");
folderParent.setParents(Arrays.asList(new ParentReference().setId("root")));
folderParent.setLabels(new Labels().setHidden(false));
driveService.files().list().execute();
folderParent = driveService.files().insert(folderParent).execute();
System.out.println("folder made with id = " + folderParent.getId());
/*
* store the parent folder id in a parent array for use by each child file
*/
List<ParentReference> parents = new ArrayList<ParentReference>();
parents.add(new ParentReference().setId(folderParent.getId()));
/*
* loop for each file
*/
for (fileCount = 0; fileCount < numFiles; fileCount++) {
/*
* make a File object for the insert
*/
File file = new File();
file.setTitle("testfile-" + (fileCount+1));
file.setParents(parents);
file.setDescription("description");
file.setMimeType("text/html");
try {
System.out.println("making file "+fileCount + " of "+numFiles);
// call the drive service insert execute method
driveService.files().insert(file).setConvert(false).execute();
} catch (GoogleJsonResponseException e) {
GoogleJsonError error = e.getDetails();
// look for rate errors and count them. Normally one would expo-backoff here, but this is to demonstrate that despite
// the 403, the file DID get created
if (error.getCode() == 403 && error.getMessage().toLowerCase().contains("rate limit")) {
System.out.println("rate limit exception on file " + fileCount + " of "+numFiles);
// increment a count of rate limit errors
ratelimitCount++;
} else {
// just in case there is a different exception thrown
System.out.println("[DbSA465] Error message: " + error.getCode() + " " + error.getMessage());
}
}
}
/*
* all done. get the children of the folder to see how many files were actually created
*/
ChildList children = driveService.children().list(folderParent.getId()).execute();
/*
* and the winner is ...
*/
System.out.println("\nSummary...");
System.out.println("File insert attempts (a) = " + numFiles);
System.out.println("rate limit errors (b) = " + ratelimitCount);
System.out.println("expected number of files (a-b) = " + (numFiles - ratelimitCount));
System.out.println("Actual number of files = " + children.getItems().size() + " NB. There is a limit of 100 children in a single page, so if you're expecting more than 100, need to follow nextPageToken");
}
}
解决方案
I'm assuming you're trying to do Parallel downloads...
This may not be an answer you're looking for, but this is what I've experienced in my interactions with google drive api. I use C#, so it's a bit different, but maybe it'll help.
I had to set a specific amount of threads to run at one time. If I let my program run all 100 entries at one time as separate threads, I run into the rate limit error as well.
I don't know well at all, but in my C# program, I run 3 threads (definable by the user, 3 is default)
opts = new ParallelOptions { MaxDegreeOfParallelism = 3 };
var checkforfinished =
Parallel.ForEach(lstBackupUsers.Items.Cast<ListViewItem>(), opts, name => {
{ // my logic code here }
I did a quick search and found that Java 8 (not sure if that's what you're using) supports Parallel().forEach(), maybe that'd help you. The resource I found for this is at: http://radar.oreilly.com/2015/02/java-8-streams-api-and-parallelism.html
Hope this helps, taking my turns trying to help others on SO as people have helped me!
这篇关于插入403的速率限制有时会成功的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
