429 请求太多 - Angular 7 - 多文件上传 [英] 429 Too Many Requests - Angular 7 - on multiple file upload

问题描述

当我尝试同时上传数百个以上的文件时，我遇到了这个问题.

I have this problem when I try to upload more than a few hundred of files at the same time.

API 接口仅用于一个文件，因此我必须调用发送每个文件的服务.现在我有这个:

The API interface is for one file only so I have to call the service sending each file. Right now I have this:

onFilePaymentSelect(event): void {
    if (event.target.files.length > 0) {
      this.paymentFiles = event.target.files[0];
    }
    let i = 0;
    let save = 0;
    const numFiles = event.target.files.length;
    let procesed = 0;
    if (event.target.files.length > 0) {
      while (event.target.files[i]) {
      const formData = new FormData();
      formData.append('file', event.target.files[i]);
      this.payrollsService.sendFilesPaymentName(formData).subscribe(
      (response) => {
        let added = null;
        procesed++;
        if (response.status_message === 'File saved') {
          added = true;
          save++;
        } else {
          added = false;
        }
        this.payList.push({ filename, message, added });
      });
    i++;
  }
}

所以我真的有一段时间将每个文件发送到 API，但我收到消息429 请求过多"，涉及大量文件.有什么办法可以改进吗?

So really I have a while for sending each file to the API but I get the message "429 too many request" on a high number of files. Any way I can improve this?

推荐答案

使用 observables 将使该任务更容易推理(而不是使用命令式编程).

Working with observables will make that task easier to reason about (rather than using imperative programming).

浏览器通常允许您并行发出 6 个请求，并将其他请求排入队列.但是我们不希望浏览器为我们管理该队列(或者如果我们在节点环境中运行，我们就不会拥有它).

A browser usually allows you to make 6 request in parallel and will queue the others. But we don't want the browser to manage that queue for us (or if we're running in a node environment we wouldn't have that for ex).

我们想要什么:我们想要上传很多文件.它们应该通过始终并行运行 5 个请求来尽可能高效地排队和上传.(因此我们为应用中的其他请求保留 1 个免费).

What do we want: We want to upload a lot of files. They should be queued and uploaded as efficiently as possible by running 5 requests in parallel at all time. (so we keep 1 free for other requests in our app).

为了演示，让我们先构建一些模拟:

In order to demo that, let's build some mocks first:

function randomInteger(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

const mockPayrollsService = {
  sendFilesPaymentName: (file: File) => {
    return of(file).pipe(
      // simulate a 500ms to 1.5s network latency from the server
      delay(randomInteger(500, 1500))
    );
  }
};

// array containing 50 files which are mocked
const files: File[] = Array.from({ length: 50 })
  .fill(null)
  .map(() => new File([], ""));

我认为上面的代码是不言自明的.我们正在生成模拟，以便我们可以看到代码的核心将如何实际运行，而无需真正访问您的应用程序.

I think the code above is self explanatory. We are generating mocks so we can see how the core of the code will actually run without having access to your application for real.

现在，主要部分:

const NUMBER_OF_PARALLEL_CALLS = 5;

const onFilePaymentSelect = (files: File[]) => {
  const uploadQueue$ = from(files).pipe(
    map(file => mockPayrollsService.sendFilesPaymentName(file)),
    mergeAll(NUMBER_OF_PARALLEL_CALLS)
  );

  uploadQueue$
    .pipe(
      scan(nbUploadedFiles => nbUploadedFiles + 1, 0),
      tap(nbUploadedFiles =>
        console.log(`${nbUploadedFiles}/${files.length} file(s) uploaded`)
      ),
      tap({ complete: () => console.log("All files have been uploaded") })
    )
    .subscribe();
};

onFilePaymentSelect(files);

• 我们使用from将文件一个一个发送到一个observable中
• 使用 map，我们准备了对 1 个文件的请求(但由于我们没有订阅它并且 observable 是冷的，请求只是准备好了，没有触发！)
• 我们现在使用 mergeMap 来运行调用池.由于 mergeMap 将并发性作为参数，我们可以说请同时运行最多 5 个调用"
• 然后我们使用 scan 仅用于显示目的(计算已成功上传的文件数)
• We use from to send the files one by one into an observable
• using map, we prepare our request for 1 file (but as we don't subscribe to it and the observable is cold, the request is just prepared, not triggered!)
• we now use mergeMap to run a pool of calls. Thanks to the fact that mergeMap takes the concurrency as an argument, we can say "please run a maximum of 5 calls at the same time"
• we then use scan for display purpose only (to count the number of files that have been uploaded successfully)

现场演示:https://stackblitz.com/edit/rxjs-zuwy33?file=index.ts

打开控制台看看我们不是一次上传所有的

Open up the console to see that we're not uploading all them at once

这篇关于429 请求太多 - Angular 7 - 多文件上传的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！