“节流"F#中的异步下载 [英] "Throttled" async download in F#

问题描述

我正在尝试下载从我博客的 xml 备份中引用的 3000 多张照片.我遇到的问题是，如果只有其中一张照片不再可用，整个异步都会被阻止，因为 AsyncGetResponse 不会超时.

I'm trying to download the 3000+ photos referenced from the xml backup of my blog. The problem I came across is that if just one of those photos is no longer available, the whole async gets blocked because AsyncGetResponse doesn't do timeouts.

ildjarn 帮助我组合了一个 AsyncGetResponse 版本，该版本在超时时会失败，但使用它可以带来很多 more 超时 - 好像只是排队超时的请求.似乎所有 WebRequest 都立即"启动，使其工作的唯一方法是将超时设置为下载所有所需的时间:这不是很好，因为这意味着我已经根据图像数量调整了超时时间.

ildjarn helped me to put together a version of AsyncGetResponse which does fail on timeout, but using that gives a lot more timeouts - as though requests that are just queued timeout. It seems like all the WebRequests are launched 'immediately', the only way to make it work is to set the timeout to the time required to download all of them combined: which isn't great because it means I have adjust the timeout depending on the number of images.

我是否达到了普通 async 的极限?我应该看看反应式扩展吗?

Have I reached the limits of vanilla async? Should I be looking at reactive extensions instead?

这有点尴尬，因为我已经问过两个 <关于这一特定代码的 href="https://stackoverflow.com/questions/5713330/how-to-handle-httpwebrequest-timeout-in-f-async-parallel">问题，我仍然没有按照我想要的方式工作！

This is a bit embarassing, because I've already asked two questions here on this particular bit of code, and I still haven't got it working the way I want!

推荐答案

我认为必须有比使用超时更好的方法来发现文件不可用.我不太确定，但是如果找不到文件，有没有办法让它抛出异常?然后你可以把你的 async 代码包装在 try .. with 中，你应该避免大多数问题.

I think there must be a better way to find out that a file is not available than using a timeout. I'm not exactly sure, but is there some way to make it throw an exception if a file cannot be found? Then you could just wrap your async code inside try .. with and you should avoid most of the problems.

无论如何，如果您想编写自己的并发管理器"来并行运行一定数量的请求并将剩余的待处理请求排队，那么 F# 中最简单的选择是使用代理(MailboxProcessor类型).以下对象封装了行为:

Anyway, if you want to write your own "concurrency manager" that runs certain number of requests in parallel and queues remaining pending requests, then the easiest option in F# is to use agents (the MailboxProcessor type). The following object encapsulates the behavior:

type ThrottlingAgentMessage = 
  | Completed
  | Work of Async<unit>

/// Represents an agent that runs operations in concurrently. When the number
/// of concurrent operations exceeds 'limit', they are queued and processed later
type ThrottlingAgent(limit) = 
  let agent = MailboxProcessor.Start(fun agent -> 
    /// Represents a state when the agent is blocked
    let rec waiting () = 
      // Use 'Scan' to wait for completion of some work
      agent.Scan(function
        | Completed -> Some(working (limit - 1))
        | _ -> None)
    /// Represents a state when the agent is working
    and working count = async { 
      while true do
        // Receive any message 
        let! msg = agent.Receive()
        match msg with 
        | Completed -> 
            // Decrement the counter of work items
            return! working (count - 1)
        | Work work ->
            // Start the work item & continue in blocked/working state
            async { try do! work 
                    finally agent.Post(Completed) }
            |> Async.Start
            if count < limit then return! working (count + 1)
            else return! waiting () }
    working 0)      

  /// Queue the specified asynchronous workflow for processing
  member x.DoWork(work) = agent.Post(Work work)

这篇关于“节流"F#中的异步下载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！