使用Jquery BlueImp上传器上传大文件后延迟 [英] Delay after uploading large files with Jquery BlueImp uploader

问题描述

我可以通过BlueImp Jquery Uploader插件成功上传大文件（测试高达8GB）。

I'm able to successfully upload large files (tested up to 8GB) via the BlueImp Jquery Uploader plugin.

我的PHP设置是：

upload_max_filesize 8192M
post_max_size 8192M
max_execution_time 200
max_input_time 200
memory_limit 8192M

我遇到的问题是，当大文件（2gb或更大）完成上传时，进度条会挂起100％并且需要相当长的时间来完成。
除了将文件块拼凑在一起之外，我不确定发生了什么后期处理。

The problem I'm having is that when large files (2gb or larger) finish uploading, the progress bar hangs on 100% and takes considerable time to "finish". I'm not sure what post processing is happening, other than piecing the file chunks together.

我已经尝试调整UploadHandler.php文件中的块大小，并且当增加时（例如从10mb到500mb块）似乎稍微改善了一些东西但延迟了还在那里。完全禁用分块上传（0）时似乎没有任何改进，我也不确定这样做的潜在后果。

I've tried adjusting the chunk size in the UploadHandler.php file, and it seems to improve things slightly when increased (e.g from 10mb to 500mb chunks) but the delay is still there. There doesn't seem to be any improvement when disabling chunked uploads altogether (0), and I'm not sure of the potential ramifications of doing this also.

2gb文件，延迟大约是20秒，但是在4gb文件上，它大概是2分钟。 7gb文件大约需要3-4分钟，有时会超时。这使用户等待，因为进度条此时已达到100％，因此不知道发生了什么。

On a 2gb file, the delay is around 20 seconds, but on a 4gb file, it's around 2 minutes. A 7gb file takes around 3-4 minutes, and sometimes times out. This leaves the user waiting, unaware of what is happening as the progress bar has finished at 100% by this point.

有没有人知道这可能是什么，或者如何进行故障排除呢？我怀疑/ tmp中的文件可能被复制而不是移动，但据我所知，在php文件中没有这个迹象。

Does anyone have any insight as to what this might be, or how to go about troubleshooting it? I suspected that the file in /tmp might be being copied rather than moved, but there's no sign of this in the php file as far as I can tell.

在我的服务器虚拟机上提升CPU和RAM可以稍微改善一点，但在此过程中运行top命令会发现CPU和RAM似乎没有广泛用尽（0-2％，相比实际上传期间的90-100％）。

Boosting the CPU and RAM on my server VM improves things a little, though running the "top" command during this process reveals that CPU and RAM don't appear to be extensively exhausted (0-2%, compared to 90-100% during actual upload).

非常感谢提前。

推荐答案

经过对此问题的大量故障排除和调试后，我发现了我认为的原因，以及更好的解决方法/解决方案。它有点hacky（我是一个业余开发者！），所以我愿意接受任何改进的建议，虽然它似乎对我有用。

After much troubleshooting and debugging of this issue, I’ve found what I believe to be the cause, and a better workaround/solution. It’s a bit "hacky" (I’m an amateur dev!), so I’m open to any suggestions to improve this, though it does seem to work for me.

从客户端上传的文件实际上传到服务器上的临时位置：
/tmp/systemd-private-random-number-httpd.service-random- number / tmp / sess_ php-session-ID

The files uploaded from the client are actually uploaded to this temp location on the server: /tmp/systemd-private-random-number-httpd.service-random-number/tmp/sess_php-session-ID

在客户端完成上传后，UI进度条当服务器处理文件时，达到100％并保持不变。

Once the uploads have completed on the client side, the UI progress bar reaches 100% and remains so whilst the server is processing the files.

服务器端的处理涉及从上面的/ tmp移动（复制，然后删除）文件位于相关的blueimp位置。 （假设blueimp选项中的user_dirs已启用​​/为true，则默认目标为：/ var / www / html / server / php / files / php-session-id /）。

The processing on the server side involves moving (copying, then deleting) the file from the above /tmp location to the relevant blueimp location. (Assuming "user_dirs" is enabled/true in the blueimp options, the default destination is: /var/www/html/server/php/files/php-session-id/ ).

此复制过程可能需要很长时间，特别是对于大于2gb或大约2gb的文件。服务器处理完成后，blueimp会触发fileuploaddone回调，并且UI会更新为已完成状态。

This copying process can take a significant amount of time, particularly on files larger than 2gb or thereabouts. Once the server processing has completed, blueimp triggers the "fileuploaddone" callback, and the UI updates to a completed state.

我的目标是在此处提供一些交互式UI反馈点（而不是像其他解决方法一样悬挂在90％）。我的环境能够进行非常大的上传（10gb +），所以我觉得不能提供任何用户反馈，因为用户认为网站已经崩溃，关闭浏览器等等。 。）。

My aim was to provide some interactive UI feedback at this point (rather than hanging at 90% like the other workaround). My environment is capable of very large uploads (10gb+), so I didn’t find it acceptable not to provide any user feedback for what could be several minutes of file processing (with the user thinking the site has crashed, and closing the browser etc.).

我的解决方法：

我遇到的第一个问题是当文件上传在客户端完成并且服务器处理开始时，似乎没有关于该点的blueimp回调。所以我通过创建一个函数（在main.js中）来解决这个问题，以便在data.loaded匹配data.total之后显示我的自定义处理div。

The first problem I encountered was that there doesn’t appear to be a blueimp callback for the point when the file uploads have completed on the client side, and the server processing begins. So I worked around this by creating a function (in main.js) to display my custom "processing div" after data.loaded matches data.total:

$('#fileupload').bind('fileuploadprogressall', function (e, data) { console.log(data); 
if (data.loaded == data.total) { 
setTimeout(function(){
$("#processwarn").fadeTo(300, 1);
},3000);
}})

我的自定义div位于进度条信息下方是另一个每隔几秒刷新一次的页面（使用ajax.load）并计算当前会话上传文件夹中所有文件的总文件大小（目录总大小似乎没有提供准确的结果）：

My custom div that sits below the progress bar info is another php page that’s refreshed (with ajax.load) every couple of seconds and calculates the total file size of all the files in the current session upload folder (the directory total size didn’t seem to provide accurate results):

// Calculate size of files being processed
$filePath = "/var/www/html/server/php/files/*php-session-id*/";

$total = 0;
$d = new RecursiveIteratorIterator(
  new RecursiveDirectoryIterator($filePath), 
  RecursiveIteratorIterator::SELF_FIRST
);

foreach($d as $file){
  $total += $file->getSize();
}

// Convert to readable format
if ($total >= 1073741824)
{
    $total = number_format($total / 1073741824, 2) . ' GB';
}
elseif ($total >= 1048576)
{
    $total = number_format($total / 1048576, 2) . ' MB';
}
elseif ($total >= 1024)
{
    $total = number_format($total / 1024, 2) . ' KB';
}
elseif ($total > 1)
{
    $total = $total . ' bytes';
}
elseif ($total == 1)
{
    $total = $total . ' byte';
}
else
{
    $total = '0 bytes';
}

// Display spinner gif and size of file currently being processed
echo "<img src=\"img/spinner.gif\" height=\"20px\" width=\"20px\">&nbsp;&nbsp;Please wait, processing files: &nbsp;&nbsp;$total";

结果如下所示：

(经过一些工作后） - 改为添加了引导程序栏：

(after some more work) - With bootstrap progress bar added instead:

测试中注意到的其他要点：

启用分块后（例如设置为1gb（maxChunkSize：1000000000）），问题几乎消失，因为处理时间显着减少到用户，因为复制处理发生在分块的时候（在这个例子中每1gb）。
当最后的处理发生时，服务器只需要重新安排/复制最后剩余的1gb。

由于这个原因我也经历了更快的整体上传时间。
事后看来，这可能是许多人更容易/更有效的解决方案。

With chunking enabled (for example set to 1gb (maxChunkSize: 1000000000) ), the problem almost disappears, as the processing time appears drastically reduced to the user, as the "copying" processing occurs at the point of chunking (every 1gb in this example). When the final processing then occurs, the server only has to "rechunk"/copy the last remaining 1gb over.
I also experienced quicker overall upload times due to this. In hindsight, this may well be an easier/more effective solution for many.

这篇关于使用Jquery BlueImp上传器上传大文件后延迟的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！