以多部分上传方式上传blob导致500mb后Chrome上的net :: ERR_FILE_NOT_FOUND [英] Uploading blobs as Multi-part upload causes net::ERR_FILE_NOT_FOUND on Chrome after 500mb
问题描述
我只在Google Chrome和Chromium上遇到了一个非常奇怪的问题。
背景是：
我使用多部分上传方法将文件上传到我的服务器，这意味着我将这些文件分成10mb的块，并将每个块发送到服务器。这可以在任何大小的文件都完美的工作，这个问题开始时，我需要加密每个块。
为了加密我使用CryptoJS，并且在上传块之前，我对它进行加密并得到 Blob 的上传结果，当我必须上传少于50个区块（50个区块，总共大约500mb）时，我得到一个 POST http：//（...）net :: ERR_FILE_NOT_FOUND 。
奇怪的是，它适用于所有其他浏览器，包括现在基本上是Chrome的Opera，除了Chrome和Chromium之外。我在IE，Firefox，Edge，Safari，Opera，Chrome和Chromium上测试过它。下面你可以看到我的代码是如何工作的，所以你们可以有一个想法，是不是我在应用程序中使用的真正的代码，而是，我写的测试代码产生相同的结果。
不是获取切片（ File.slice ）我将上传为一个块并加密它以获得 blob ，我是会用我的块的大小生成一个假的 blob 。我把 setTimeout 模拟加密blob所花费的时间。就像我之前说过的，通过这样做，我得到了和真实代码一样的结果：
function uploadNext（prevResponse）{
if（currentPart == totalPartsFile）
return;
// var chunk = getNextChunk（）;
var totalSize = file.size;
$ b $ setTimeout（function（）{
var blob = new Blob（[new ArrayBuffer（constants.chunkSize）]，{
type：'application / octet-string'，
名称：file.name
}）;
console.log（blob）;
blob.encrypted = true;
blob。 key = encryptionKey;
blob.mimeType = file.mimeType;
blob.name = file.name;
blob.originalFileSize = originalFileSize || file.size;
）uploadFile（objectId，currentPart，blob，totalSize，prevResponse，function（resp）{
uploadNext（resp）;
}）;
}，1000）;
$ b因此，上面的代码是我的blob生成的地方，下面有上传部分：
function uploadFile（objectId，index，blob，totalSize，prevResponse，callback）{
var format =encrypted;
var params =？format =+ format +（format ===encrypted？& encoding = base64：）;
var endPoint = constants.contentServiceUrl + resourceService.availableResources.addContents.link.split（'：objectId'）。join（objectId）+ params;
var formData = new FormData（）;
formData.append（totalFileSizeBytes，totalSize）;
formData.append（partIndex，index）;
formData.append（partByteOffset，previousOffset）;
formData.append（chunkSize，blob.size）;
formData.append（totalParts，totalPartsFile）;
formData.append（filename，blob.name）;
if（currentPart！= 0）{
formData.append（uploadId，prevResponse.uploadId）;
formData.append（bucket，prevResponse.bucket）;
}
if（finalChunk）{
for（var key in etags1）{
formData.append（etags [+ key +]，etags1 [键]）;
}
}
formData.append（data，blob）;
PreviousOffset + = blob.size;
var request = {
method：'POST'，
url：endPoint，
data：formData，
headers：{
' （函数（d））{
_.extend（etags1，d.etags）;
console.log（d）;
callback（d）;
}）
.error（function（d）{
console.log（d）;
}）;
}
当然还有其他支持变量和代码，我没有放在这里，但这足以说明我们正在处理的事情。
在这个例子中，我使用了AngularJS' $ http 模块，但我尝试过使用纯粹的 XMLHttpRequest ，并且得到了相同的结果。
正如我所说的，我只获得了大于499mb（50+块）以上的文件的 POST POST http：//（...）net :: ERR_FILE_NOT_FOUND 。在这里发布这个，因为我一直在寻找一个解决方案，但我找不到任何与这个问题有关的东西，我发现最接近的东西在互联网上是Chromium项目论坛中的这个问题：
https://code.google.com/p/chromium/issues/detail?id=375297
此时我真的不知道该怎么办了，所以我想知道是否
感谢您提前给出答案。
解决方案在最后的铬源文件中，我发现了一个blob限制。
-
ChromeOS：
- Ram - 20％
- 磁盘 - 50％注意：磁盘是用户分区，所以如果这个操作系统已满，操作系统仍然可以正常工作。
-
Android：
- 内存 - 1％
- 磁盘 - 6％
- 桌面：
- Ram - 20％，如果是x64，则为2 GB。 - 10％
铬回购链接： https://cs.chromium.org/chromium/src/storage/browser/ blob / blob_memory_controller.cc？l = 63
I'm having a really weird issue only on Google Chrome and Chromium.
The background is:
I upload files to my server using the multi-part upload method, meaning that I break the files into chunks of 10mb and send each chunk to the server. This works flawlessly in all browsers with files of any size, the issue started when I needed to encrypt each chunk.
For encryption I use CryptoJS and, before uploading the chunk, I encrypt it and get the resulting Blob to upload, this works fine on Chrome when I have to upload less than 50 chunks (50 blobs, around 500mb in total), after that I get a POST http://(...) net::ERR_FILE_NOT_FOUND.
Weirdly, this works on all of the other browsers, including Opera which is basically Chrome nowadays, except Chrome and Chromium. I tested it on IE, Firefox, Edge, Safari, Opera, Chrome and Chromium.
Below you can see how my code works so you guys can have an idea, this is not the real code I use in the app but, rather, it's a test code I wrote that yields the same result.
Instead of getting a slice (File.slice) of the File I'm going to upload as a chunk and encrypting it to get the blob, I'm going to generate a bogus blob with the size of my chunk. I put the setTimeout to simulate the time it takes to encrypt a blob. Like I said before, I get the same result as my real code by doing this:
function uploadNext(prevResponse) {
if (currentPart == totalPartsFile)
return;
//var chunk = getNextChunk();
var totalSize = file.size;
setTimeout(function() {
var blob = new Blob([new ArrayBuffer(constants.chunkSize)], {
type: 'application/octet-string',
name: file.name
});
console.log(blob);
blob.encrypted = true;
blob.key = encryptionKey;
blob.mimeType = file.mimeType;
blob.name = file.name;
blob.originalFileSize = originalFileSize || file.size;
uploadFile(objectId, currentPart, blob, totalSize, prevResponse, function(resp) {
uploadNext(resp);
});
}, 1000);
}
So, the code above is where my blob is generated, below there's the upload part:
function uploadFile (objectId, index, blob, totalSize, prevResponse, callback) {
var format = "encrypted";
var params = "?format=" + format + (format === "encrypted" ? "&encoding=base64" : "");
var endPoint = constants.contentServiceUrl + resourceService.availableResources.addContents.link.split(':objectId').join(objectId) + params;
var formData = new FormData();
formData.append("totalFileSizeBytes", totalSize);
formData.append("partIndex", index);
formData.append("partByteOffset", previousOffset);
formData.append("chunkSize", blob.size);
formData.append("totalParts", totalPartsFile);
formData.append("filename", blob.name);
if (currentPart != 0) {
formData.append("uploadId", prevResponse.uploadId);
formData.append("bucket", prevResponse.bucket);
}
if (finalChunk) {
for (var key in etags1) {
formData.append("etags[" + key + "]", etags1[key]);
}
}
formData.append("data", blob);
previousOffset += blob.size;
var request = {
method: 'POST',
url: endPoint,
data: formData,
headers: {
'Content-Type': 'multipart/form-data'
}
}
$http(request)
.success(function(d) {
_.extend(etags1, d.etags);
console.log(d);
callback(d);
})
.error(function(d) {
console.log(d);
});
}
Of course there are other supporting variables and code that I didn't put here, but this is enough to give an idea of what we're dealing with.
In this example I'm using AngularJS' $http module, but I've tried with pure XMLHttpRequest as well and I got the same result.
Like I said, I only get the POST http://(...) net::ERR_FILE_NOT_FOUND with files bigger than 499mb (50+ chunks) and only in Chrome.
I'm posting this here as I've been looking for a solution but I couldn't find anything related to this problem, the closest thing I found on the internet was this issue in the Chromium project forum:
https://code.google.com/p/chromium/issues/detail?id=375297
At this point I really don't know what to do anymore so I'd like to know if anyone has had a similar problem in the past and could fix it somehow.
Thank you for the answers in advance.
解决方案 At last chromium source files, I had found a blob limits.
ChromeOS:
- Ram - 20%
- Disk - 50% Note: The disk is the user partition, so the operating system can still function if this is full.
Android:
- RAM - 1%
- Disk - 6%
- Desktop:
- Ram - 20%, or 2 GB if x64.
- Disk - 10%
chromium repo link: https://cs.chromium.org/chromium/src/storage/browser/blob/blob_memory_controller.cc?l=63
这篇关于以多部分上传方式上传blob导致500mb后Chrome上的net :: ERR_FILE_NOT_FOUND的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
