429请求过多-Angular 7-上载多个文件 [英] 429 Too Many Requests - Angular 7 - on multiple file upload

问题描述

当我尝试同时上传数百个文件时，我遇到了这个问题.

I have this problem when I try to upload more than a few hundred of files at the same time.

API接口仅用于一个文件，因此我必须调用发送每个文件的服务.现在我有这个:

The API interface is for one file only so I have to call the service sending each file. Right now I have this:

onFilePaymentSelect(event): void {
    if (event.target.files.length > 0) {
      this.paymentFiles = event.target.files[0];
    }
    let i = 0;
    let save = 0;
    const numFiles = event.target.files.length;
    let procesed = 0;
    if (event.target.files.length > 0) {
      while (event.target.files[i]) {
      const formData = new FormData();
      formData.append('file', event.target.files[i]);
      this.payrollsService.sendFilesPaymentName(formData).subscribe(
      (response) => {
        let added = null;
        procesed++;
        if (response.status_message === 'File saved') {
          added = true;
          save++;
        } else {
          added = false;
        }
        this.payList.push({ filename, message, added });
      });
    i++;
  }
}

所以我确实有一段时间将每个文件发送到API，但是我收到大量文件的消息"429太多请求".有什么我可以改善的方法吗?

So really I have a while for sending each file to the API but I get the message "429 too many request" on a high number of files. Any way I can improve this?

推荐答案

使用可观察对象将使该任务更易于推理(而不是使用命令式编程).

Working with observables will make that task easier to reason about (rather than using imperative programming).

浏览器通常允许您并行发出6个请求，并将其他请求排队.但是我们不希望浏览器为我们管理该队列(或者，如果我们在节点环境中运行，那么我们就不会拥有该队列).

A browser usually allows you to make 6 request in parallel and will queue the others. But we don't want the browser to manage that queue for us (or if we're running in a node environment we wouldn't have that for ex).

我们想要什么:我们要上传很多文件.通过始终并行运行5个请求，应尽可能高效地对它们进行排队和上传. (因此我们为应用程序中的其他请求保留了1个免费空间).

What do we want: We want to upload a lot of files. They should be queued and uploaded as efficiently as possible by running 5 requests in parallel at all time. (so we keep 1 free for other requests in our app).

为了演示它，让我们先构建一些模拟:

In order to demo that, let's build some mocks first:

function randomInteger(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

const mockPayrollsService = {
  sendFilesPaymentName: (file: File) => {
    return of(file).pipe(
      // simulate a 500ms to 1.5s network latency from the server
      delay(randomInteger(500, 1500))
    );
  }
};

// array containing 50 files which are mocked
const files: File[] = Array.from({ length: 50 })
  .fill(null)
  .map(() => new File([], ""));

我认为上面的代码是不言自明的.我们正在生成模拟，因此我们可以看到代码的核心将如何真正运行，而无需真正访问您的应用程序.

I think the code above is self explanatory. We are generating mocks so we can see how the core of the code will actually run without having access to your application for real.

现在，主要部分:

const NUMBER_OF_PARALLEL_CALLS = 5;

const onFilePaymentSelect = (files: File[]) => {
  const uploadQueue$ = from(files).pipe(
    map(file => mockPayrollsService.sendFilesPaymentName(file)),
    mergeAll(NUMBER_OF_PARALLEL_CALLS)
  );

  uploadQueue$
    .pipe(
      scan(nbUploadedFiles => nbUploadedFiles + 1, 0),
      tap(nbUploadedFiles =>
        console.log(`${nbUploadedFiles}/${files.length} file(s) uploaded`)
      ),
      tap({ complete: () => console.log("All files have been uploaded") })
    )
    .subscribe();
};

onFilePaymentSelect(files);

• 我们使用from将文件一一发送到一个可观察的
• 使用map，我们准备1个文件的请求(但由于我们未订阅该文件，并且可观察到的状态很冷，因此该请求只是准备就绪，未触发！)
• 我们现在使用mergeMap来运行呼叫池.由于mergeMap将并发作为参数，因此我们可以说请同时运行最多5个调用"
• 然后我们将scan仅用于显示目的(计算已成功上传的文件数)
• We use from to send the files one by one into an observable
• using map, we prepare our request for 1 file (but as we don't subscribe to it and the observable is cold, the request is just prepared, not triggered!)
• we now use mergeMap to run a pool of calls. Thanks to the fact that mergeMap takes the concurrency as an argument, we can say "please run a maximum of 5 calls at the same time"
• we then use scan for display purpose only (to count the number of files that have been uploaded successfully)

这是一个实时演示: https://stackblitz.com/edit/rxjs-zuwy33?file=index.ts

Here's a live demo: https://stackblitz.com/edit/rxjs-zuwy33?file=index.ts

打开控制台，看看我们没有一次全部上传

Open up the console to see that we're not uploading all them at once

这篇关于429请求过多-Angular 7-上载多个文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！