大量下载文件 [英] Downloading a file in multiple chunks

问题描述

我正在编写一段应该以1 MB的块下载单个文件的代码.我使用与此问题相同的代码:在C#中的小块并将其转换为VB.NET.

代码与上一个问题相同，可以正常工作并将文件写入磁盘.但是似乎在第二个Web请求中出了点问题.为了进行测试，我从 https://d13yacurqjgara.cloudfront.net/users/22/screenshots/631004/attachments/53012/wallpaper-retina-cinemadisplay.png 使用下面的VB.NET代码.

下载的PNG似乎是正确的一半，剩下的图像被打乱了.

我将 defaultSize 和 chunk 增大到10 MB，文件下载完美，但是似乎For/Next循环中的某些内容会截断或污染数据./p>

有什么想法会导致这种情况发生?

 私有常量defaultSize as Long = 1048576专用数据块，只要= 1048576私人补偿，只要Long = 0私有函数downloadFile(ByVal url作为字符串，ByVal文件名作为字符串)作为布尔值尽可能长的Dim大小= getSize(url)Dim blockSize As Integer = Convert.ToInt32(大小/defaultSize)余数为Integer = Convert.ToInt32(size Mod defaultSize)如果余数>0然后blockSize + = 1万一昏暗的FileStream为FileStream = File.Create("C:\ mydirectory \"& filename)对于i As Integer = 0到blockSize-1如果i = blockSize-1然后块=余数万一Dim req As HttpWebRequest = HttpWebRequest.Create(url)req.Method = WebRequestMethods.Http.Getreq.AddRange(Convert.ToInt32(offset)，Convert.ToInt32(chunk + offset))Dim resp As HttpWebResponse = req.GetResponse()使用respStream作为Stream = resp.GetResponseStream昏暗缓冲区(4096)以字节为单位昏暗字节读取为整数做bytesRead = respStream.Read(缓冲区，0，4096)如果bytesRead>0然后fileStream.Write(buffer，0，bytesRead)字节读取>时循环0最终使用偏移量+ =块resp.Close()resp.Dispose()下一个fileStream.Close()返回真结束功能私有函数getSize(ByVal url As String)只要很长暗淡的要求为WebRequest = WebRequest.Create(url)req.Method = WebRequestMethods.Http.Head暗淡作为WebResponse = req.GetResponse返回Long.Parse(resp.ContentLength)结束功能 

解决方案

除了整理一些代码(我想您还是会这样做，因为您是从C#转换过来的)，只需做一点小改动就可以了完美地工作.

行

  offset + =块 

应该是

  offset + =块+ 1 

这解决了这个问题，因为 offset 从0开始，并且您要在下一个循环中向其添加块大小，因此您要添加一个兆字节并从此处开始.您要从最后一个兆字节之后的字节继续！我确实开始解释了它是如何工作的，但是我想出的示例不起作用...所以我无法解释它！也许其他人可以；与 req.AddRange 有关-您指定的是范围，而不是总数.

I'm writing a piece of code that is supposed to download a single file in 1 MB chunks. I used the same code as in this question: Download large file in small chunks in C# and converted it to VB.NET.

The code, same as the previous question works fine and write the file to disk. But it seems that something goes wrong in the second web request. For the sake of testing, I'm downloading a PNG file from https://d13yacurqjgara.cloudfront.net/users/22/screenshots/631004/attachments/53012/wallpaper-retina-cinemadisplay.png using my VB.NET code below.

The downloaded PNG seems to be fine half the way and the remaining image is scrambled.

I increased the defaultSize and chunk to 10 MB and the file downloads perfectly, but it seems that something in the For/Next loop truncates or pollutes the data.

Any thoughts what might be causing this to happen?

Private Const defaultSize As Long = 1048576
Private chunk As Long = 1048576
Private offset As Long = 0

Private Function downloadFile(ByVal url As String, ByVal filename As String) As Boolean
    Dim size As Long = getSize(url)
    Dim blockSize As Integer = Convert.ToInt32(size / defaultSize)
    Dim remainder As Integer = Convert.ToInt32(size Mod defaultSize)

    If remainder > 0 Then
        blockSize += 1
    End If

    Dim fileStream As FileStream = File.Create("C:\mydirectory\" & filename)

    For i As Integer = 0 To blockSize - 1
        If i = blockSize - 1 Then
            chunk = remainder
        End If

        Dim req As HttpWebRequest = HttpWebRequest.Create(url)
        req.Method = WebRequestMethods.Http.Get
        req.AddRange(Convert.ToInt32(offset), Convert.ToInt32(chunk + offset))
        Dim resp As HttpWebResponse = req.GetResponse()

        Using respStream As Stream = resp.GetResponseStream
            Dim buffer(4096) As Byte
            Dim bytesRead As Integer
            Do
                bytesRead = respStream.Read(buffer, 0, 4096)
                If bytesRead > 0 Then fileStream.Write(buffer, 0, bytesRead)
            Loop While bytesRead > 0
        End Using

        offset += chunk

        resp.Close()
        resp.Dispose()
    Next

    fileStream.Close()

    Return True
End Function

Private Function getSize(ByVal url As String) As Long
    Dim req As WebRequest = WebRequest.Create(url)
    req.Method = WebRequestMethods.Http.Head
    Dim resp As WebResponse = req.GetResponse
    Return Long.Parse(resp.ContentLength)
End Function

解决方案

Aside from tidying up a bit of your code (which I presume you will be doing anyway, since you converted this from C#), making one small change gets it work perfectly.

The line

offset += chunk

should be

offset += chunk + 1

This solves the issue because offset starts at 0, and you're adding the chunk size to it on the next loop, therefore you're adding a megabyte and starting there. You want to be continuing from the byte following the last megabyte! I did begin to explain how it works, but the example I came up with doesn't work... so I can't explain it! Maybe someone else can; it's to do with req.AddRange - you're specifying a range, not a total.

这篇关于大量下载文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！