将 blob 作为多部分上传上传会在 500mb 后导致 Chrome 上的 net::ERR_FILE_NOT_FOUND [英] Uploading blobs as Multi-part upload causes net::ERR_FILE_NOT_FOUND on Chrome after 500mb

问题描述

我只在 Google Chrome 和 Chromium 上遇到了一个非常奇怪的问题.

I'm having a really weird issue only on Google Chrome and Chromium.

背景是:

我使用分段上传方法将文件上传到我的服务器，这意味着我将文件分成 10mb 的块并将每个块发送到服务器.这在具有任何大小文件的所有浏览器中都可以完美运行，当我需要加密每个块时问题就开始了.

I upload files to my server using the multi-part upload method, meaning that I break the files into chunks of 10mb and send each chunk to the server. This works flawlessly in all browsers with files of any size, the issue started when I needed to encrypt each chunk.

对于加密，我使用 CryptoJS，在上传块之前，我对其进行加密并获取生成的 Blob 上传，当我必须上传少于 50 个块(50 个 blob)时，这在 Chrome 上工作正常，总共大约 500mb)，之后我得到一个 POST http://(...) net::ERR_FILE_NOT_FOUND.

For encryption I use CryptoJS and, before uploading the chunk, I encrypt it and get the resulting Blob to upload, this works fine on Chrome when I have to upload less than 50 chunks (50 blobs, around 500mb in total), after that I get a POST http://(...) net::ERR_FILE_NOT_FOUND.

奇怪的是，这适用于所有其他浏览器，包括现在基本上是 Chrome 的 Opera，除了 Chrome 和 Chromium.我在 IE、Firefox、Edge、Safari、Opera、Chrome 和 Chromium 上对其进行了测试.

Weirdly, this works on all of the other browsers, including Opera which is basically Chrome nowadays, except Chrome and Chromium. I tested it on IE, Firefox, Edge, Safari, Opera, Chrome and Chromium.

您可以在下面看到我的代码是如何工作的，所以你们可以有一个想法，这不是我在应用程序中使用的真实代码，而是我编写的产生相同结果的测试代码.

Below you can see how my code works so you guys can have an idea, this is not the real code I use in the app but, rather, it's a test code I wrote that yields the same result.

我不是获取文件的切片 (File.slice)，而是将其作为块上传并对其进行加密以获取 blob，而是将生成一个具有我的块大小的虚假 blob .我把 setTimeout 用来模拟加密 blob 所需的时间.就像我之前说的，通过这样做，我得到了与真实代码相同的结果:

Instead of getting a slice (File.slice) of the File I'm going to upload as a chunk and encrypting it to get the blob, I'm going to generate a bogus blob with the size of my chunk. I put the setTimeout to simulate the time it takes to encrypt a blob. Like I said before, I get the same result as my real code by doing this:

function uploadNext(prevResponse) {  
    if (currentPart == totalPartsFile)
        return;

    //var chunk = getNextChunk();
    var totalSize = file.size;

    setTimeout(function() {
        var blob = new Blob([new ArrayBuffer(constants.chunkSize)], {
            type: 'application/octet-string',
            name: file.name
        });

        console.log(blob);

        blob.encrypted = true;
        blob.key = encryptionKey;
        blob.mimeType = file.mimeType;
        blob.name = file.name;
        blob.originalFileSize = originalFileSize || file.size;
        
        uploadFile(objectId, currentPart, blob, totalSize, prevResponse, function(resp) {
            uploadNext(resp);
        });
    }, 1000);
}

所以，上面的代码是生成我的 blob 的地方，下面是上传部分:

So, the code above is where my blob is generated, below there's the upload part:

function uploadFile (objectId, index, blob, totalSize, prevResponse, callback) {

    var format = "encrypted";
    var params = "?format=" + format + (format === "encrypted" ? "&encoding=base64" : "");
    var endPoint = constants.contentServiceUrl + resourceService.availableResources.addContents.link.split(':objectId').join(objectId) + params;

    var formData = new FormData();

    formData.append("totalFileSizeBytes", totalSize);
    formData.append("partIndex", index);
    formData.append("partByteOffset", previousOffset);
    formData.append("chunkSize", blob.size);
    formData.append("totalParts", totalPartsFile);
    formData.append("filename", blob.name);

    if (currentPart != 0) {
        formData.append("uploadId", prevResponse.uploadId);
        formData.append("bucket", prevResponse.bucket);
    }

    if (finalChunk) {
        for (var key in etags1) {
            formData.append("etags[" + key + "]", etags1[key]);
        }
    }

    formData.append("data", blob);

    previousOffset += blob.size;

    var request = {
        method: 'POST',
        url: endPoint,
        data: formData,
        headers: {
            'Content-Type': 'multipart/form-data'
        }
    }

    $http(request)
        .success(function(d) {
            _.extend(etags1, d.etags);
            console.log(d);
            callback(d);
        })
        .error(function(d) {
        console.log(d);
    });                                                
}

当然还有其他支持变量和代码我没有放在这里，但这足以让我们了解我们正在处理的内容.

Of course there are other supporting variables and code that I didn't put here, but this is enough to give an idea of what we're dealing with.

在这个例子中，我使用了 AngularJS 的 $http 模块，但我也尝试过使用纯 XMLHttpRequest 并且得到了相同的结果.

In this example I'm using AngularJS' $http module, but I've tried with pure XMLHttpRequest as well and I got the same result.

就像我说的，我只得到 POST http://(...) net::ERR_FILE_NOT_FOUND 文件大于 499mb(50+ 块)并且仅在 Chrome 中.

Like I said, I only get the POST http://(...) net::ERR_FILE_NOT_FOUND with files bigger than 499mb (50+ chunks) and only in Chrome.

我在这里发布此问题是因为我一直在寻找解决方案，但我找不到与此问题相关的任何内容，我在互联网上找到的最接近的内容是 Chromium 项目论坛中的此问题:

I'm posting this here as I've been looking for a solution but I couldn't find anything related to this problem, the closest thing I found on the internet was this issue in the Chromium project forum:

https://code.google.com/p/chromium/问题/详细信息?id=375297

在这一点上，我真的不知道该怎么办了，所以我想知道过去是否有人遇到过类似的问题并且可以以某种方式解决它.

At this point I really don't know what to do anymore so I'd like to know if anyone has had a similar problem in the past and could fix it somehow.

感谢您提前回答.

推荐答案

终于找到了 chromium 源文件，发现了一个 blob 限制.

At last chromium source files, I had found a blob limits.

1. Chrome 操作系统:

1. ChromeOS:

• 内存 - 20%
• 磁盘 - 50% 注意:磁盘是用户分区，因此如果已满，操作系统仍然可以运行.

安卓:

• 内存 - 1%
• 磁盘 - 6%

• 内存 - 20%，如果 x64 则为 2 GB.
• 磁盘 - 10%

铬回购链接:https:///cs.chromium.org/chromium/src/storage/browser/blob/blob_memory_controller.cc?l=63

这篇关于将 blob 作为多部分上传上传会在 500mb 后导致 Chrome 上的 net::ERR_FILE_NOT_FOUND的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！