＆QUOT;＆宋长福QUOT;在F＃异步下载 [英] "Throttled" async download in F#

问题描述

我想下载3000+的照片从我的博客的XML备份引用。我碰到的问题是，如果只是这些照片之一，是不再可用，整个异步被阻止，因为AsyncGetResponse不做超时。

I'm trying to download the 3000+ photos referenced from the xml backup of my blog. The problem I came across is that if just one of those photos is no longer available, the whole async gets blocked because AsyncGetResponse doesn't do timeouts.

ildjarn 帮我把一个版本AsyncGetResponse那里没有失败的超时时间，但使用，让很多更多的超时 - 似乎这只是排队超时请求。好像一切都WebRequests推出的立即，使之工作的唯一方法是将超时设置为所需的时间来下载的所有的人联合：这是不是很大，因为这意味着我有根据图像的数量调整超时。

ildjarn helped me to put together a version of AsyncGetResponse which does fail on timeout, but using that gives a lot more timeouts - as though requests that are just queued timeout. It seems like all the WebRequests are launched 'immediately', the only way to make it work is to set the timeout to the time required to download all of them combined: which isn't great because it means I have adjust the timeout depending on the number of images.

我是否达到了香草的异步的限制？我应该在无扩展找呢？

Have I reached the limits of vanilla async? Should I be looking at reactive extensions instead?

的这是一个有点尴尬，因为我已经问 2 <一个href=\"http://stackoverflow.com/questions/5713330/how-to-handle-httpwebrequest-timeout-in-f-async-parallel\">questions这里code这个特殊的位，我仍然没有得到它的工作就是我想要的！的

This is a bit embarassing, because I've already asked two questions here on this particular bit of code, and I still haven't got it working the way I want!

推荐答案

我想一定要找出一个文件是不是比使用超时提供一个更好的办法。我不完全确定，但有一些方法，使之抛出一个异常，如果一个文件无法找到？然后，你可以只换你的异步 code里面的尝试..以键，你应该避免的大部分问题

I think there must be a better way to find out that a file is not available than using a timeout. I'm not exactly sure, but is there some way to make it throw an exception if a file cannot be found? Then you could just wrap your async code inside try .. with and you should avoid most of the problems.

无论如何，如果你想写运行若干并行剩余未决的处理请求和队列自己的并发管理器，然后在F＃最简单的方法是使用代理（即 MailboxProcessor ​​类型）。以下对象封装的行为：

Anyway, if you want to write your own "concurrency manager" that runs certain number of requests in parallel and queues remaining pending requests, then the easiest option in F# is to use agents (the MailboxProcessor type). The following object encapsulates the behavior:

type ThrottlingAgentMessage = 
  | Completed
  | Work of Async<unit>

/// Represents an agent that runs operations in concurrently. When the number
/// of concurrent operations exceeds 'limit', they are queued and processed later
type ThrottlingAgent(limit) = 
  let agent = MailboxProcessor.Start(fun agent -> 
    /// Represents a state when the agent is blocked
    let rec waiting () = 
      // Use 'Scan' to wait for completion of some work
      agent.Scan(function
        | Completed -> Some(working (limit - 1))
        | _ -> None)
    /// Represents a state when the agent is working
    and working count = async { 
      while true do
        // Receive any message 
        let! msg = agent.Receive()
        match msg with 
        | Completed -> 
            // Decrement the counter of work items
            return! working (count - 1)
        | Work work ->
            // Start the work item & continue in blocked/working state
            async { try do! work 
                    finally agent.Post(Completed) }
            |> Async.Start
            if count < limit then return! working (count + 1)
            else return! waiting () }
    working 0)      

  /// Queue the specified asynchronous workflow for processing
  member x.DoWork(work) = agent.Post(Work work)

这篇关于＆QUOT;＆宋长福QUOT;在F＃异步下载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！