在rails应用程序中处理大型文件上传的最佳方法是什么？ [英] What is the best approach to handle large file uploads in a rails app?

问题描述

我有兴趣了解在Rails应用程序中处理大型文件上传的不同方法，2-5Gb文件。

I am interested in understanding the different approaches to handling large file uploads in a Rails application, 2-5Gb files.

我理解为了传输文件这个尺寸需要分解成更小的部分，我做了一些研究，这是我到目前为止所做的。

I understand that in order to transfer a file of this size it will need to be broken down into smaller parts, I have done some research and here is what I have so far.

• 服务器端配置将需要接受大型POST请求，并且可能需要64位机器来处理。


• AWS支持分段上传。


• HTML5 FileSystemAPI 有一个持久的上传器，可以按块上传文件。


• Bitorrent 虽然这需要一个不理想的传输客户端

• Server-side config will be required to accept large POST requests and probably a 64bit machine to handle anything over 4Gb.
• AWS supports multipart upload.
• HTML5 FileSystemAPI has a persistent uploader that uploads the file in chunks.
• A library for Bitorrent although this requires a transmission client which is not ideal

所有这些方法都能像FTP一样恢复，我不想使用FTP的原因是，如果可能的话，我想保留在网络应用程序中？我使用了carrierwave和paperclip，但我正在寻找可以恢复的东西，因为上传一个5Gb的文件可能需要一些时间！

Can all of these methods be resumed like FTP, the reason I dont want to use FTP is that I want to keep in the web app if this is possible? I have used carrierwave and paperclip but I am looking for something that will be able to be resumed as uploading a 5Gb file could take some time!

我列出的这些方法我想了解哪些方法运作良好，如果还有其他方法可能会丢失？如果可能的话，没有插件，宁愿不使用Java Applets或Flash。另一个问题是这些解决方案在上传时将文件保存在内存中，这也是我希望尽可能避免的约束。

Of these approaches I have listed I would like to undertand what has worked well and if there are other approaches that I may be missing? No plugins if possible, would rather not use Java Applets or Flash. Another concern is that these solutions hold the file in memory while uploading, that is also a constraint I would rather avoid if possible.

推荐答案

我已经在几个网站上处理了这个问题，使用了上面介绍的一些技术和一些你没有说过的技术。好消息是允许大量上传实际上是非常现实的。

I've dealt with this issue on several sites, using a few of the techniques you've illustrated above and a few that you haven't. The good news is that it is actually pretty realistic to allow massive uploads.

这很大程度上取决于你上传文件后实际计划对文件做些什么...你需要做的工作就越多，文件，离你越近，你想要它到你的服务器。如果您需要立即处理上传，您可能想要做一个纯粹的rails解决方案。如果您不需要进行任何处理，或者它不是时间关键的，您可以开始考虑混合解决方案...

A lot of this depends on what you actually plan to do with the file after you have uploaded it... The more work you have to do on the file, the closer you are going to want it to your server. If you need to do immediate processing on the upload, you probably want to do a pure rails solution. If you don't need to do any processing, or it is not time-critical, you can start to consider "hybrid" solutions...

信不信由你我使用 mod_porter 实际上运气不错。 Mod_porter让apache做了一堆你的应用通常会做的工作。它有助于在上传过程中不占用线程和一堆内存。它会在您的应用中生成一个本地文件，以便于处理。如果你注意处理上传文件的方式（思考流），你可以使整个过程使用非常少的内存，即使是传统上相当昂贵的操作。这种方法需要很少的实际设置才能使您的应用程序正常工作，并且不需要对代码进行真正的修改，但它确实需要特定的环境（apache服务器），以及配置它的能力。

Believe it or not, I've actually had pretty good luck just using mod_porter. Mod_porter makes apache do a bunch of the work that your app would normally do. It helps not tie up a thread and a bunch of memory during the upload. It results in a file local to your app, for easy processing. If you pay attention to the way you are processing the uploaded files (think streams), you can make the whole process use very little memory, even for what would traditionally be fairly expensive operations. This approach requires very little actual setup to your app to get working, and no real modification to your code, but it does require a particular environment (apache server), as well as the ability to configure it.

我也很幸运使用 jQuery-File-上传，支持chunked和resumable uploads等好东西。如果没有类似mod_porter的东西，这仍然可以在上传过程中占用整个执行线程，但如果操作正确，它应该在内存上合适。这也会导致文件关闭，因此易于处理。这种方法需要调整您的视图层才能实现，并且不适用于所有浏览器。

I've also had good luck using jQuery-File-Upload, which supports good stuff like chunked and resumable uploads. Without something like mod_porter, this can still tie up an entire thread of execution during upload, but it should be decent on memory, if done right. This also results in a file that is "close" and, as a result, easy to process. This approach will require adjustments to your view layer to implement, and will not work in all browsers.

您提到了FTP和bittorrent作为可能的选项。这些并不像你想象的那么糟糕，因为你仍然可以让文件非常靠近服务器。它们甚至不是互斥的，这很好，因为（正如你所指出的）它们确实需要一个额外的客户端，可能会或可能不会出现在上传机器上。它的工作方式基本上就是为你设置一个可以转储的区域，你的应用程序可以看到它。然后，如果您需要进行任何处理，则运行一个cron作业（或其他）来监视该位置以进行上载并触发您的服务器处理方法。这并不能让您获得上述方法可以提供的即时响应，但您可以将间隔设置得足够小以使其非常接近。这种方法唯一真正的优点是所使用的协议更适合传输大文件，根据我的经验，额外的客户需求和分散的流程通常会超过任何好处。

You mentioned FTP and bittorrent as possible options. These are not as bad of options as you might think, as you can still get the files pretty close to the server. They are not even mutually exclusive, which is nice, because (as you pointed out) they do require an additional client that may or may not be present on the uploading machine. The way this works is, basically, you set up an area for them to dump to that is visible by your app. Then, if you need to do any processing, you run a cron job (or whatever) to monitor that location for uploads and trigger your servers processing method. This does not get you the immediate response the methods above can provide, but you can set the interval to be small enough to get pretty close. The only real advantage to this method is that the protocols used are better suited to transferring large files, the additional client requirement and fragmented process usually outweigh any benefits from that, in my experience.

如果您根本不需要任何处理，最好的选择可能是直接与他们一起使用S3。这个解决方案是你实际需要对除了将它们作为静态资产服务器之外的文件做任何事情的第二个....

If you don't need any processing at all, your best bet may be to simply go straight to S3 with them. This solution falls down the second you actually need to do anything with the files other than server them as static assets....

我没有任何使用HTML5的经验在rails应用程序中的FileSystemAPI，所以我不能说到这一点，虽然它似乎会严重限制你能够支持的客户。

I do not have any experience using the HTML5 FileSystemAPI in a rails app, so I can't speak to that point, although it seems that it would significantly limit the clients you are able to support.

不幸的是，那里不是一个真正的银弹 - 所有这些选项都需要在你想要完成的环境中与你的环境进行权衡。例如，您可能无法配置Web服务器或永久写入本地文件系统。对于它的价值，我认为jQuery-File-Upload可能是您在大多数环境中最好的选择，因为它只需要修改您的应用程序，因此您可以最轻松地将实现移动到另一个环境。

Unfortunately, there is not one real silver bullet - all of these options need to be weighed against your environment in the context of what you are trying to accomplish. You may not be able to configure your web server or permanently write to your local file system, for example. For what it's worth, I think jQuery-File-Upload is probably your best bet in most environments, as it only really requires modification to your application, so you could move an implementation to another environment most easily.

这篇关于在rails应用程序中处理大型文件上传的最佳方法是什么？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！