创建并流式传输大型存档，而不将其存储在内存或磁盘上 [英] Create and stream a large archive without storing it in memory or on disk

问题描述

我想允许用户一次下载多个大文件的存档。但是，文件和存档可能太大而无法存储在我的服务器的内存或磁盘上（它们可以动态地从其他服务器流入）。我希望在将文件传输给用户时生成存档。

I want to allow users to download an archive of multiple large files at once. However, the files and the archive may be too large to store in memory or on disk on my server (they are streamed in from other servers on the fly). I'd like to generate the archive as I stream it to the user.

我可以使用Tar或Zip或其他最简单的方法。我正在使用django，它允许我在我的响应中返回一个生成器或类文件对象。该对象可用于泵送过程。但是，我无法弄清楚如何围绕zipfile或tarfile库构建这种东西，我担心它们可能不支持在文件读取时读取文件，或者在构建时读取存档。

I can use Tar or Zip or whatever is simplest. I am using django, which allows me to return a generator or file-like object in my response. This object could be used to pump the process along. However, I am having trouble figuring out how to build this sort of thing around the zipfile or tarfile libraries, and I'm afraid they may not support reading files as they go, or reading the archive as it is built.

将迭代器转换为类文件对象可能有所帮助。 tarfile #addfile 需要一个可迭代的，但它似乎立即传递给 shutil.copyfileobj ，所以这可能不是像我希望的那样对生成器友好。

This answer on converting an iterator to a file-like object might help. tarfile#addfile takes an iterable, but it appears to immediately pass that to shutil.copyfileobj, so this may not be as generator-friendly as I had hoped.

推荐答案

我最终使用 SpiderOak ZipStream 。

这篇关于创建并流式传输大型存档，而不将其存储在内存或磁盘上的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！