部分下载在C＃序列化的大文件？ [英] Partially download and serialize big file in C#?
问题描述
由于我大学即将到来的项目的一部分，我需要写一个客户端从服务器下载媒体文件，并将其写入到本地磁盘。由于这些文件可能非常大，我需要实现，以避免过多的内存使用部分下载和系列化
我想出什么样的主意：
命名空间PartialDownloadTester
{$ b $使用系统b;使用System.Diagnostics.Contracts
;
：使用System.IO;使用System.Net
;
使用System.Text;
公共类DownloadClient
{
公共静态无效的主要（字串[] args）
{
变种DLC =新DownloadClient（参数[0] ARGS [1]，ARGS [2]）;
dlc.DownloadAndSaveToDisk（）;
到Console.ReadLine（）;
}
私人的WebRequest请求;
//文件
私人字符串目录的目录;
//完整的文件标识符
私人字符串文件路径;
公共DownloadClient（URI字符串，字符串文件名，字符串的fileType）
{
this.request = WebRequest.Create（URI）;
this.request.Method =GET;
变种某人=新的StringBuilder（）;
sb.Append（C：\\testdata\\DownloadedData\\）;
this.dir = sb.ToString（）;
sb.Append（。文件名++的fileType）;
this.filePath = sb.ToString（）;
}
公共无效DownloadAndSaveToDisk（）
{
//确保目录存在
this.CreateDir（）;
VAR响应=（HttpWebResponse）request.GetResponse（）;
Console.WriteLine（内容长度：+ response.ContentLength）;
VAR rStream = response.GetResponseStream（）;
INT读取动作= -1;
做
{
变种BUF =新的字节[2048];
读取动作= rStream.Read（BUF，0，buf.Length）;
rStream.Flush（）;
this.SerializeFileChunk（BUF）;
}
，而（读取动作！= 0）;
}
私人无效CreateDir（）
{
如果（！Directory.Exists（DIR））
{
Directory.CreateDirectory（ DIR）;
}
}
私人无效SerializeFileChunk（字节[]字节）
{
Contract.Requires（！Object.ReferenceEquals（字节，NULL）） ;
的FileStream FS = File.Open（文件路径，FileMode.Append）;
fs.Write（字节，0，bytes.Length）;
fs.Flush（）;
fs.Close（）;
}
}
}
有关测试的目的，我已经使用以下参数：
http://itu.dk/people/janv/mufc_abc.jpgmufc_abc，JPG格式
然而，该图片是不完整的（只有第一〜10％外表右），即使内容长度打印63780这是，图像的实际尺寸
所以我的问题是：
- 这是去部分下载和序列化的正确方法，或有更好的/更简单的方法？
- 存储在客户端内存响应流中的全部内容？如果是这样的话，做我需要使用HttpWebRequest.AddRange部分地从服务器下载数据，以节省我的客户端的内存？
- 为什么序列化失败，我得到一个破碎形象？
- 请我介绍了不少的开销，当我使用FileMode.Append？ （MSDN指出，这一选项搜索到文件的结尾）
在此先感谢
我不知道这个问题的根源，但我会改变这样的循环
const int的CHUNKSIZE = 2048;
变种BUF =新的字节[CHUNKSIZE]
VAR rStream = response.GetResponseStream（）;
做{
INT读取动作= rStream.Read（BUF，0，CHUNKSIZE）;
如果（读取动作大于0）{
this.SerializeFileChunk（BUF，读取动作）;
}
}，而（读取动作== CHUNKSIZE）;
序列化方法会得到一个额外的参数
私人无效SerializeFileChunk（字节[]字节，诠释的numBytes）
然后写字节
fs.Write（字节，0的numBytes）的权数;
更新：
我没有看到结束的需要，每次重新打开该文件。我也使用使用语句，关闭资源，即使要发生异常。在使用语句调用的Dispose（）末资源的方法，这反过来又调用关闭（）文件流的情况。 使用可用于实施的IDisposable 。
<各类pre>
VAR BUF =新的字节[2048];
使用（VAR rStream = response.GetResponseStream（））{使用（的FileStream FS = File.Open（文件路径，FileMode.Append））{
做{
读取动作= rStream .Read（BUF，0，buf.Length）;
fs.Write（字节，0，读取动作）;
}，而（...）;
}
}
using语句做这样的事情。
{
VAR rStream = response.GetResponseStream（）;
试
{
//做rStream一些工作在这里。
} {最后
如果（rStream！= NULL）{
rStream.Dispose（）;
}
}
}
As part of an upcoming project at my university, I need to write a client that downloads a media file from a server and writes it to the local disk. Since these files can be very large, I need to implement partial download and serialization in order to avoid excessive memory use.
What I came up with:
namespace PartialDownloadTester
{
using System;
using System.Diagnostics.Contracts;
using System.IO;
using System.Net;
using System.Text;
public class DownloadClient
{
public static void Main(string[] args)
{
var dlc = new DownloadClient(args[0], args[1], args[2]);
dlc.DownloadAndSaveToDisk();
Console.ReadLine();
}
private WebRequest request;
// directory of file
private string dir;
// full file identifier
private string filePath;
public DownloadClient(string uri, string fileName, string fileType)
{
this.request = WebRequest.Create(uri);
this.request.Method = "GET";
var sb = new StringBuilder();
sb.Append("C:\\testdata\\DownloadedData\\");
this.dir = sb.ToString();
sb.Append(fileName + "." + fileType);
this.filePath = sb.ToString();
}
public void DownloadAndSaveToDisk()
{
// make sure directory exists
this.CreateDir();
var response = (HttpWebResponse)request.GetResponse();
Console.WriteLine("Content length: " + response.ContentLength);
var rStream = response.GetResponseStream();
int bytesRead = -1;
do
{
var buf = new byte[2048];
bytesRead = rStream.Read(buf, 0, buf.Length);
rStream.Flush();
this.SerializeFileChunk(buf);
}
while (bytesRead != 0);
}
private void CreateDir()
{
if (!Directory.Exists(dir))
{
Directory.CreateDirectory(dir);
}
}
private void SerializeFileChunk(byte[] bytes)
{
Contract.Requires(!Object.ReferenceEquals(bytes, null));
FileStream fs = File.Open(filePath, FileMode.Append);
fs.Write(bytes, 0, bytes.Length);
fs.Flush();
fs.Close();
}
}
}
For testing purposes, I've used the following parameters:
"http://itu.dk/people/janv/mufc_abc.jpg" "mufc_abc" "jpg"
However, the picture is incomplete (only the first ~10% look right) even though the content length prints 63780 which is the actual size of the image.
So my questions are:
- Is this the right way to go for partial download and serialization or is there a better/easier approach?
- Is the full content of the response stream stored in client memory? If this is the case, do I need to use HttpWebRequest.AddRange to partially download data from the server in order to conserve my client's memory?
- How come the serialization fails and I get a broken image?
- Do I introduce a lot of overhead when I use the FileMode.Append? (msdn states that this option "seeks to the end of the file")
Thanks in advance
I don't know if this the source of the problem, however I would change the loop like this
const int ChunkSize = 2048;
var buf = new byte[ChunkSize];
var rStream = response.GetResponseStream();
do {
int bytesRead = rStream.Read(buf, 0, ChunkSize);
if (bytesRead > 0) {
this.SerializeFileChunk(buf, bytesRead);
}
} while (bytesRead == ChunkSize);
The serialize method would get an additional argument
private void SerializeFileChunk(byte[] bytes, int numBytes)
and then write the right number of bytes
fs.Write(bytes, 0, numBytes);
UPDATE:
I do not see the need for closing and reopening the file each time. I also would use the using statement, which closes the resources, even if an exception should occur. The using statement calls the Dispose() method of the resource at the end, which in turn calls Close() in the case of file streams. using can be applied to all types implementing IDisposable.
var buf = new byte[2048];
using (var rStream = response.GetResponseStream()) {
using (FileStream fs = File.Open(filePath, FileMode.Append)) {
do {
bytesRead = rStream.Read(buf, 0, buf.Length);
fs.Write(bytes, 0, bytesRead);
} while (...);
}
}
The using statement does something like this
{
var rStream = response.GetResponseStream();
try
{
// do some work with rStream here.
} finally {
if (rStream != null) {
rStream.Dispose();
}
}
}
这篇关于部分下载在C＃序列化的大文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
