在Haskell中从Internet下载大文件 [英] Downloading large files from the Internet in Haskell

问题描述

有关于如何在Haskell中下载大文件的建议吗？我图Http.Conduit是图书馆是一个很好的图书馆。但是，它是如何解决这个问题的？在它的文档中有一个示例，但它不适合下载大文件，它只是下载文件：

  import Data.Conduit.Binary（sinkFile）
 import Network.HTTP.Conduit 
将限定的Data.Conduit导入为C 
 
 main :: IO（）
 main = do 
请求<  -  parseUrlhttp://google.com/
 withManager $ \manager  - > 
响应<  -  http请求管理器
响应Body响应C。$$ +  -  sinkFilegoogle.html
  

我想要的是能够下载大文件，而不会耗尽RAM，例如在性能等方面做到有效。最好能够继续下载它们，这意味着现在某些部分，后来另一部分。

我也是找到下载卷曲包hackage ，但我并不积极，这是一个很好的选择，甚至可以像我需要的那样通过块下载文件。

/ div>

Network.HTTP.Conduit 提供了三种执行请求的函数：

• simpleHttp


• httpLbs


• http

前两个函数将使整个响应体驻留在内存中。如果你想在常量内存中运行，那么使用 http 函数。 http 函数可让您通过 ResumableSource

您在代码中提供了使用交错IO来将响应主体写入恒定内存空间中的文件。因此，下载大文件时不会出现内存不足的情况。

Are there any suggestions about how to download large files in Haskell? I figure Http.Conduit is is the library is a good library for this. However, how does it solve this? There is an example in its documentation but it is not fit for downloading large files, it just downloads a file:

 import Data.Conduit.Binary (sinkFile)
 import Network.HTTP.Conduit
 import qualified Data.Conduit as C

 main :: IO ()
 main = do
      request <- parseUrl "http://google.com/"
      withManager $ \manager -> do
          response <- http request manager
          responseBody response C.$$+- sinkFile "google.html"

What I want is be able to download large files and not run out of RAM, e.g. do it effectively in terms of performance, etc. Preferably, being able to continue downloading them "later", meaning "some part now, another part later".

I also found the download-curl package on hackage, but I'm not positive this is a good fit, or even that it downloads files chunk by chunk like I need.

解决方案

Network.HTTP.Conduit provides three functions for performing a request:

• simpleHttp
• httpLbs
• http

Out of the three functions, the first two functions will make the entire response body to live in memory. If you want to operate in constant memory, then use http function. The http function gives you access to a streaming interface through ResumableSource

The example you have provided in your code uses interleaved IO to write the response body to a file in constant memory space. So, you will not run out of memory when downloading a large file.

这篇关于在Haskell中从Internet下载大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！