RoR - 在rails中上传大文件 [英] RoR - Large file uploads in rails

问题描述

目前的设置对于较小的文件是适用的，但我还需要支持大文件上传（高达4GB）。当我尝试上传一个4GB的文件时，它最终会发生，但从用户体验的角度来看，这是非常糟糕的：上传开始，进度显示基于XHR的进度事件，但100％后，仍然等待很长时间（5分钟）在服务器响应请求之前。

最初我以为这是将文件从临时目录复制到最终的NFS挂载目录。但现在我不太确定。在向我的路由添加日志记录之后，我发现在文件上载进度达到100％之间，以及我的控制器操作中的代码运行（在执行将文件移动到NAS之前的处理）之前，需要等待3分钟左右， 。

我想知道以下内容：

• 什么是在上传完成之后，在我的操作被调用之前，在这3分钟的等待中发生了什么？


• 有没有一种方法可以解释这段时间内正在发生的事情，客户端在上传完成后立即得到响应，以便它们不超时？


• 在Rails中通常如何处理大文件上传？这似乎是一个普遍的问题，但我似乎无法找到任何东西。

（注意：最初使用CarrierWave上传时，我发现这个问题，我删除它，只是直接在我的模型中使用FileUtils处理文件保存，以确保等待时间不是一些CarrierWave魔术幕后发生的结果，但得到）

ruby​​ -v：1.9.3p362

rails -v：3.2.11 我终于找到了我的主要问题的答案：
在这3分钟的等待期间发生了什么事？/ b> 上传完成之前，我的行动被称为？

在这篇文章中已经很清楚地解释了：
Rails方式 - 上传文件

当浏览器上传一个文件时，它会编码内容的格式称为多部分MIME（与发送电子邮件附件时使用的格式相同）。为了让你的应用程序对这个文件做些什么，rails必须撤消这个编码。要做到这一点，需要阅读巨大的请求体，并将每行与几个正则表达式进行匹配。

我尝试了 modporter 这个帖子中提到的Apache模块，唯一的问题是这个模块及其相应的插件是在4年前写的，而且他们的网站不再在操作中，几乎没有任何一个文件。

使用 modporter ，我想指定我的NFS挂载的目录作为PorterDir，希望它将文件传递到NAS，而不需要从临时目录中进行任何额外的复制。但是，我无法获得到目前为止，模块似乎忽略了我指定的PorterDir，并且正在返回一个完全不同的路径到我的动作，最重要的是它返回的路径根本不存在，所以我不知道实际发生了什么我的上传。

我的解决方法

我不得不迅速解决问题，现在它包括编写相应的JavaScript / Ruby代码，以处理分块文件上传。

JS示例：

  var MAX_CHUNK_SIZE = 20000000; //以字节为单位
 
 window.FileUploader = function（opts）{
 var file = opts.file; 
 var url = opts.url; 
 var current_byte = 0; 
 var success_callback = opts.success; 
 var progress_callback = opts.progress; 
 var percent_complete = 0; 
 
 this.start = this.resume = function（）{
 paused = false; 
 upload（）; 
}; 
 
 this.pause = function（）{
 paused = true; 
}; 
 
函数上传（）{
 var chunk = file.slice（current_byte，current_byte + MAX_CHUNK_SIZE）; 
 var fd = new FormData（）; 
 fd.append（'chunk'，chunk）; 
 fd.append（'filename'，file.name）; 
 fd.append（'total_size'，file.size）; 
 fd.append（'start_byte'，current_byte）; 
 $ b $ .ajax（url，{
 type：'post'，
 data：fd，
成功：函数（数据）{
 current_byte = data.next_byte; $ b $ upload_id = data.upload_id; 
 
 if（data.path）{
 success_callback（data.path）; 
} 
 else {
 percent_complete = Math.round（current_byte / file.size * 100）; 
 if（percent_complete> 100）percent_complete = 100; 
 progress_callback（percent_complete）; //更新UI元素向用户提供反馈
 upload（）; 
} 
} 
}）; 
} 
}; 
  

（原谅任何语法错误，只需从头开始输入）

服务器端，我创建了一个新的路由来接受文件块。在第一个块提交时，我生成一个基于文件名/大小的upload_id，并确定是否已经有一个来自中断上载的部分文件。如果是这样的话，我将ID和下一个我需要的起始字节一起传回。如果没有，我存储第一个块并传回ID。

这个过程有额外的块上传，附加部分文件，直到文件大小与原始文件大小相匹配。此时，服务器响应文件的临时路径。

然后，javascript从表单中删除文件输入，并用一个隐藏的输入替换它的值是从服务器返回的文件路径，然后张贴表单。

然后，最后在服务器端，我处理移动/重命名文件并保存其最终路径到我的模型。

唷。

I have a rails webapp that allows users to upload videos, where they are stored in an NFS-mounted directory.

The current setup is fine for smaller files, but I need to support large file uploads as well (up to 4gb). When I try to upload a 4gb file, it eventually happens but is awful from a UX standpoint: upload starts and progress is displayed based on XHR 'progress' events, but then after 100%, there is still a long wait (5+ minutes) before the server responds to the request.

Initially I thought this had to do with copying the file from some temp directory over to the final NFS-mounted directory. But now I'm not so sure. After adding logging to my routes, I see that there is about a 3-minute wait between when the file upload progress reaches 100% and when the code in my controller action runs (before I do any handling for moving the file to the NAS).

I'm wondering the following:

• What is happening during this 3 minute wait after the upload completes and before my action is called?
• Is there a way for me to account for whatever is going on during this period so that the client gets a response immediately after the upload completes so that they don't time out?
• How are large file uploads typically handled in Rails? This seems like it would be a common problem, but I can't seem to find anything on it.

(Note: I was originally using CarrierWave for uploads when I discovered this problem. I removed it and simply handled the file save using FileUtils directly in my model just to make sure the wait times weren't the result of some CarrierWave magic happening behind the scenes, but got exactly the same result.)

ruby -v: 1.9.3p362

rails -v: 3.2.11

解决方案

I finally found the answer to my main question: What is happening during this 3 minute wait after the upload completes and before my action is called?

It's all explained very clearly in this post: The Rails Way - Uploading Files

"When a browser uploads a file, it encodes the contents in a format called ‘multipart mime’ (it’s the same format that gets used when you send an email attachment). In order for your application to do something with that file, rails has to undo this encoding. To do this requires reading the huge request body, and matching each line against a few regular expressions. This can be incredibly slow and use a huge amount of CPU and memory."

I tried the modporter Apache module mentioned in the post. The only problem is that the module and its corresponding plugin were written 4 years ago, and with their website no longer in operation, there's almost no documentation on either one.

With modporter, I wanted to specify my NFS-mounted directory as the PorterDir, in the hopes that it would pass the file right along to the NAS without any extra copying from a temp directory. However, I was not able to get this far since the module seemed to be ignoring my specified PorterDir, and was returning a completely different path to my actions. On top of that, the path it was returning didn't even exist, so I had no idea what was actually happening to my uploads.

My Workaround

I had to get the problem solved quickly, so I went with a somewhat hacky solution for now which consisted of writing corresponding JavaScript/Ruby code in order to handle chunked file uploads.

JS Example:

var MAX_CHUNK_SIZE = 20000000; // in bytes

window.FileUploader = function (opts) {
    var file = opts.file;
    var url = opts.url;
    var current_byte = 0;
    var success_callback = opts.success;
    var progress_callback = opts.progress;
    var percent_complete = 0;

    this.start = this.resume = function () {
        paused = false;
        upload();
    };

    this.pause = function () {
        paused = true;
    };

    function upload() {
        var chunk = file.slice(current_byte, current_byte + MAX_CHUNK_SIZE);
        var fd = new FormData();
        fd.append('chunk', chunk);
        fd.append('filename', file.name);
        fd.append('total_size', file.size);
        fd.append('start_byte', current_byte);

        $.ajax(url, {
          type: 'post',
          data: fd,
          success: function (data) {
              current_byte = data.next_byte;
              upload_id = data.upload_id;

              if (data.path) {
                  success_callback(data.path);
              }
              else {
                  percent_complete= Math.round(current_byte / file.size * 100);
                  if (percent_complete> 100) percent_complete = 100;
                  progress_callback(percent_complete); // update some UI element to provide feedback to user
                  upload();
              }
          }
        });
    }
};

(forgive any syntax errors, just typing this off the top of my head)

Server-side, I created a new route to accept the file chunks. On first chunk submission, I generate an upload_id based on filename/size, and determine if I already have a partial file from an interrupted upload. If so, I pass back the next starting byte I need along with the id. If not, I store the first chunk and pass back the id.

The process with additional chunk uploads appending the partial file until the file size matches the original file size. At this point, the server responds with the temporary path to the file.

The javascript then removes the file input from the form, and replaces it with a hidden input whose value is the file path returned from the server, and then posts the form.

Then finally server-side, I handle moving/renaming the file and saving its final path to my model.

Phew.

这篇关于RoR - 在rails中上传大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！