下载大文件 [英] Download large file
问题描述
通过使用UnityEngine.WWW下载文件，我得到了错误
OverflowException:数字溢出.
我发现错误是由结构本身引起的，因为字节数组的字节数超过了int.MaxValue可以分配的字节数(〜2GB).
通过返回带有www.bytes的数组会引发该错误，这意味着框架可能以其他方式存储该数组.
如何以其他方式访问下载的数据，或者有更大文件的替代方法?
public IEnumerator downloadFile()
{
WWW www = new WWW(filesource);
while(!www.isDone)
{
progress = www.progress;
yield return null;
}
if(string.IsNullOrEmpty(www.error))
{
data = www.bytes; // <- Errormessage fired here
}
}
新答案(Unity 2017.2及更高版本)
将 UnityWebRequest 与 DownloadHandlerFile . DownloadHandlerFile类是新的，用于直接下载和保存文件，同时防止占用大量内存.
IEnumerator Start()
{
string url = "http://dl3.webmfiles.org/big-buck-bunny_trailer.webm";
string vidSavePath = Path.Combine(Application.persistentDataPath, "Videos");
vidSavePath = Path.Combine(vidSavePath, "MyVideo.webm");
//Create Directory if it does not exist
if (!Directory.Exists(Path.GetDirectoryName(vidSavePath)))
{
Directory.CreateDirectory(Path.GetDirectoryName(vidSavePath));
}
var uwr = new UnityWebRequest(url);
uwr.method = UnityWebRequest.kHttpVerbGET;
var dh = new DownloadHandlerFile(vidSavePath);
dh.removeFileOnAbort = true;
uwr.downloadHandler = dh;
yield return uwr.SendWebRequest();
if (uwr.isNetworkError || uwr.isHttpError)
Debug.Log(uwr.error);
else
Debug.Log("Download saved to: " + vidSavePath.Replace("/", "\\") + "\r\n" + uwr.error);
}
旧答案(Unity 2017.1及更低版本)，如果您要在下载文件时访问每个字节，请使用
像这样的问题是为什么Unity的 WWW API现在已在最新版本的Unity中的UnityWebRequest API之上实现，这意味着，如果您在WWW API上遇到错误，使用UnityWebRequest您也可能会遇到相同的错误.即使可行，您可能也会在使用Android等小内存的移动设备上遇到问题.
该怎么做，请使用UnityWebRequest的 DownloadHandlerScript 功能，该功能可让您分块下载数据.通过分块下载数据，可以防止导致溢出错误. WWW API没有实现此功能，因此必须使用UnityWebRequest和DownloadHandlerScript来分块下载数据.您可以此处.. >
这应该可以解决您当前的问题，但是在尝试使用File.WriteAllBytes保存大数据时，您可能会遇到另一个内存问题.使用FileStream进行保存，并仅在下载完成后关闭它.
创建一个自定义UnityWebRequest，以按如下方式分块下载数据:
using System;
using System.IO;
using UnityEngine;
using UnityEngine.Networking;
public class CustomWebRequest : DownloadHandlerScript
{
// Standard scripted download handler - will allocate memory on each ReceiveData callback
public CustomWebRequest()
: base()
{
}
// Pre-allocated scripted download handler
// Will reuse the supplied byte array to deliver data.
// Eliminates memory allocation.
public CustomWebRequest(byte[] buffer)
: base(buffer)
{
Init();
}
// Required by DownloadHandler base class. Called when you address the 'bytes' property.
protected override byte[] GetData() { return null; }
// Called once per frame when data has been received from the network.
protected override bool ReceiveData(byte[] byteFromServer, int dataLength)
{
if (byteFromServer == null || byteFromServer.Length < 1)
{
Debug.Log("CustomWebRequest :: ReceiveData - received a null/empty buffer");
return false;
}
//Write the current data chunk to file
AppendFile(byteFromServer, dataLength);
return true;
}
//Where to save the video file
string vidSavePath;
//The FileStream to save the file
FileStream fileStream = null;
//Used to determine if there was an error while opening or saving the file
bool success;
void Init()
{
vidSavePath = Path.Combine(Application.persistentDataPath, "Videos");
vidSavePath = Path.Combine(vidSavePath, "MyVideo.webm");
//Create Directory if it does not exist
if (!Directory.Exists(Path.GetDirectoryName(vidSavePath)))
{
Directory.CreateDirectory(Path.GetDirectoryName(vidSavePath));
}
try
{
//Open the current file to write to
fileStream = new FileStream(vidSavePath, FileMode.OpenOrCreate, FileAccess.ReadWrite);
Debug.Log("File Successfully opened at" + vidSavePath.Replace("/", "\\"));
success = true;
}
catch (Exception e)
{
success = false;
Debug.LogError("Failed to Open File at Dir: " + vidSavePath.Replace("/", "\\") + "\r\n" + e.Message);
}
}
void AppendFile(byte[] buffer, int length)
{
if (success)
{
try
{
//Write the current data to the file
fileStream.Write(buffer, 0, length);
Debug.Log("Written data chunk to: " + vidSavePath.Replace("/", "\\"));
}
catch (Exception e)
{
success = false;
}
}
}
// Called when all data has been received from the server and delivered via ReceiveData
protected override void CompleteContent()
{
if (success)
Debug.Log("Done! Saved File to: " + vidSavePath.Replace("/", "\\"));
else
Debug.LogError("Failed to Save File to: " + vidSavePath.Replace("/", "\\"));
//Close filestream
fileStream.Close();
}
// Called when a Content-Length header is received from the server.
protected override void ReceiveContentLength(int contentLength)
{
//Debug.Log(string.Format("CustomWebRequest :: ReceiveContentLength - length {0}", contentLength));
}
}
使用方法:
UnityWebRequest webRequest;
//Pre-allocate memory so that this is not done each time data is received
byte[] bytes = new byte[2000];
IEnumerator Start()
{
string url = "http://dl3.webmfiles.org/big-buck-bunny_trailer.webm";
webRequest = new UnityWebRequest(url);
webRequest.downloadHandler = new CustomWebRequest(bytes);
webRequest.SendWebRequest();
yield return webRequest;
}
By downloading a file with UnityEngine.WWW, I get the error
OverflowException: Number overflow.
I found out the error is caused form the structure itself, because the byte-array has more bytes than int.MaxValue can allocate (~2GB).
The error is fired by returning the array with www.bytes, which means, that the framework probably stores the array in on other way.
How can I access the downloaded data in another way or is there an alternative for bigger files?
public IEnumerator downloadFile()
{
WWW www = new WWW(filesource);
while(!www.isDone)
{
progress = www.progress;
yield return null;
}
if(string.IsNullOrEmpty(www.error))
{
data = www.bytes; // <- Errormessage fired here
}
}
New answer (Unity 2017.2 and above)
Use UnityWebRequest with DownloadHandlerFile. The DownloadHandlerFile class is new and is used to download and save file directly while preventing high memory usage.
IEnumerator Start()
{
string url = "http://dl3.webmfiles.org/big-buck-bunny_trailer.webm";
string vidSavePath = Path.Combine(Application.persistentDataPath, "Videos");
vidSavePath = Path.Combine(vidSavePath, "MyVideo.webm");
//Create Directory if it does not exist
if (!Directory.Exists(Path.GetDirectoryName(vidSavePath)))
{
Directory.CreateDirectory(Path.GetDirectoryName(vidSavePath));
}
var uwr = new UnityWebRequest(url);
uwr.method = UnityWebRequest.kHttpVerbGET;
var dh = new DownloadHandlerFile(vidSavePath);
dh.removeFileOnAbort = true;
uwr.downloadHandler = dh;
yield return uwr.SendWebRequest();
if (uwr.isNetworkError || uwr.isHttpError)
Debug.Log(uwr.error);
else
Debug.Log("Download saved to: " + vidSavePath.Replace("/", "\\") + "\r\n" + uwr.error);
}
OLD answer (Unity 2017.1 and below) Use if you want to access each bytes while the file is downloading)
A problem like this is why Unity's UnityWebRequest was made but it won't work directly because WWW API is now implemented on top of the UnityWebRequest API in the newest version of Unity which means that if you get error with the WWW API, you will also likely get that same error with UnityWebRequest. Even if it works, you'll likely have have issues on mobile devices with the small ram like Android.
What to do is use UnityWebRequest's DownloadHandlerScript feature which allows you to download data in chunks. By downloading data in chunks, you can prevent causing the overflow error. The WWW API did not implement this feature so UnityWebRequest and DownloadHandlerScript must be used to download the data in chunks. You can read how this works here.
While this should solve your current issue, you may run into another memory issue when trying to save that large data with File.WriteAllBytes. Use FileStream to do the saving part and close it only when the download has finished.
Create a custom UnityWebRequest for downloading data in chunks as below:
using System;
using System.IO;
using UnityEngine;
using UnityEngine.Networking;
public class CustomWebRequest : DownloadHandlerScript
{
// Standard scripted download handler - will allocate memory on each ReceiveData callback
public CustomWebRequest()
: base()
{
}
// Pre-allocated scripted download handler
// Will reuse the supplied byte array to deliver data.
// Eliminates memory allocation.
public CustomWebRequest(byte[] buffer)
: base(buffer)
{
Init();
}
// Required by DownloadHandler base class. Called when you address the 'bytes' property.
protected override byte[] GetData() { return null; }
// Called once per frame when data has been received from the network.
protected override bool ReceiveData(byte[] byteFromServer, int dataLength)
{
if (byteFromServer == null || byteFromServer.Length < 1)
{
Debug.Log("CustomWebRequest :: ReceiveData - received a null/empty buffer");
return false;
}
//Write the current data chunk to file
AppendFile(byteFromServer, dataLength);
return true;
}
//Where to save the video file
string vidSavePath;
//The FileStream to save the file
FileStream fileStream = null;
//Used to determine if there was an error while opening or saving the file
bool success;
void Init()
{
vidSavePath = Path.Combine(Application.persistentDataPath, "Videos");
vidSavePath = Path.Combine(vidSavePath, "MyVideo.webm");
//Create Directory if it does not exist
if (!Directory.Exists(Path.GetDirectoryName(vidSavePath)))
{
Directory.CreateDirectory(Path.GetDirectoryName(vidSavePath));
}
try
{
//Open the current file to write to
fileStream = new FileStream(vidSavePath, FileMode.OpenOrCreate, FileAccess.ReadWrite);
Debug.Log("File Successfully opened at" + vidSavePath.Replace("/", "\\"));
success = true;
}
catch (Exception e)
{
success = false;
Debug.LogError("Failed to Open File at Dir: " + vidSavePath.Replace("/", "\\") + "\r\n" + e.Message);
}
}
void AppendFile(byte[] buffer, int length)
{
if (success)
{
try
{
//Write the current data to the file
fileStream.Write(buffer, 0, length);
Debug.Log("Written data chunk to: " + vidSavePath.Replace("/", "\\"));
}
catch (Exception e)
{
success = false;
}
}
}
// Called when all data has been received from the server and delivered via ReceiveData
protected override void CompleteContent()
{
if (success)
Debug.Log("Done! Saved File to: " + vidSavePath.Replace("/", "\\"));
else
Debug.LogError("Failed to Save File to: " + vidSavePath.Replace("/", "\\"));
//Close filestream
fileStream.Close();
}
// Called when a Content-Length header is received from the server.
protected override void ReceiveContentLength(int contentLength)
{
//Debug.Log(string.Format("CustomWebRequest :: ReceiveContentLength - length {0}", contentLength));
}
}
How to use:
UnityWebRequest webRequest;
//Pre-allocate memory so that this is not done each time data is received
byte[] bytes = new byte[2000];
IEnumerator Start()
{
string url = "http://dl3.webmfiles.org/big-buck-bunny_trailer.webm";
webRequest = new UnityWebRequest(url);
webRequest.downloadHandler = new CustomWebRequest(bytes);
webRequest.SendWebRequest();
yield return webRequest;
}
这篇关于下载大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
