如何在处理速率限制时异步发送 HTTP 请求? [英] How can I send HTTP Requests asynchronously while handling rate-limits?
问题描述
免责声明:我是 sttp 和 Monix,这是我尝试了解更多关于这些库的尝试.我的目标是通过 HTTP GET 请求从给定的 API 获取数据(客户端)->解析 JSON 响应 ->将此信息写入数据库.我的问题仅与第一部分有关.我的目标是以异步(希望是快速的)方式运行 get 请求,同时有办法避免或处理速率限制.
Disclaimer: I am new to sttp and Monix, and that is my attempt to learn more about these libraries. My goal is to fetch data (client-side) from a given API via HTTP GET requests -> parse JSON responses -> write this information to a database. My question pertains to the first part only. My objective is to run get requests in an asynchronous (hopefully fast) way while having a way to either avoid or handle rate limits.
以下是我已经尝试过的片段,似乎适用于单个请求:
Below is a snippet of what I have already tried, and seems to work for a single request:
package com.github.client
import io.circe.{Decoder, HCursor}
import sttp.client._
import sttp.client.circe._
import sttp.client.asynchttpclient.monix._
import monix.eval.Task
object SO extends App {
case class Bla(paging: Int)
implicit val dataDecoder: Decoder[Bla] = (hCursor: HCursor) => {
for {
next_page <- hCursor.downField("foo").downArray.downField("bar").as[Int]
} yield Bla(next_page)
}
val postTask = AsyncHttpClientMonixBackend().flatMap { implicit backend =>
val r = basicRequest
.get(uri"https://foo.bar.io/v1/baz")
.header("accept", "application/json")
.header("Authorization", "hushh!")
.response(asJson[Bla])
r.send() // How can I instead of operating on a single request, operate on multiple
.flatMap { response =>
Task(response.body)
}
.guarantee(backend.close())
}
import monix.execution.Scheduler.Implicits.global
postTask.runSyncUnsafe() match {
case Left(error) => println(s"Error when executing request: $error")
case Right(data) => println(data)
}
}
我的问题:
- 如何通过使用 Monix 处理多个 GET 请求(而不是单个请求),同时保持代码异步和可组合
- 如何避免或处理由 api 服务器强加的命中率限制
顺便说一句,如果支持速率限制目标,我在使用另一个后端方面也很灵活.
On a side note, I am also flexible in terms of using another back-end if that will support the rate limiting objective.
推荐答案
你可以像这样使用 monix.reactive.Observable
You can use monix.reactive.Observable like this
Observable.repeatEval(postTask) // we generate infinite observable of the task
.throttle(1.second, 3) // set throttling
.mapParallelOrderedF(2)(_.runToFuture) // set execution parallelism and execute tasks
.subscribe() // start the pipline
while (true) {}
这篇关于如何在处理速率限制时异步发送 HTTP 请求?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!