通过响应流发送大文件 [英] Send big file over reactive stream

问题描述

我正在编写的部分应用程序需要从客户端到服务器任意传输大文件（对于这个问题，我假设100-200 GB）。重要的是，接收方（服务器）没有存储此文件-它只是读取/检查流并将其发送到下一个点。因为我根本不需要整个文件，但是希望同时进行多个传输，所以我想最大程度地减少RAM使用率并消除磁盘​​使用率。我想处理1 MB的文件。

现在，服务器使用Spring Boot和Akka。

<我的第一个尝试是在客户端打开缓冲的文件输入流，以1 MB的块读取它，然后在单独的线程中以消息形式发送它们。它起作用了，但是问题是客户端正在一个接一个地发送消息，而不用担心服务器是否有缓冲区来存储它（缺少背压）。

我的第二个想法是使用这样的akka​​流：

如何使用反应性流进行NIO二进制处理？

通过使用ActorPublisher像这样：

带有akka群集的akka​​-streams

，但是，如此处所述：

http://doc.akka.io/docs/akka/2.4.16/scala/stream /stream-integrations.html#Implementing_Reactive_Streams_Publisher_or_Subscriber

警告
A未来版本的Akka中可能不建议使用ctorPublisher和ActorSubscriber。

警告
ActorPublisher和ActorSubscriber不能与远程actor一起使用，因为如果响应流的信号协议（例如请求）将会丢失，否则流可能会死锁。

这看起来不是个好主意。

我不想将其保存在任何存储提供程序（Dropbox，Google驱动器等）中，因为我想即时分析数据。我已经安装了Spring 5和Akka，但是我可以使用任何其他软件，

主要问题是：如何流式传输假设服务器无法一次在磁盘上或ram中存储文件，那么从客户端到服务器的大文件？

奖金问题是：如何计算正确的文件大小

我一直在寻找答案好几天了，而且看起来我不是唯一遇到这种问题的人，但是没有答案或答案就像不要做一样，没有指出其他适当的替代解决方案。

解决方案

Akka流提供了功能专门针对此用例：流式传输文件IO 。从文档中：

  import akka.stream.scaladsl._ 
 val file = Paths.get（ example。 csv）
 
 val foreach：Future [IOResult] = 
 FileIO.fromPath（file）
 .to（Sink.ignore）
 .run（）
  

关于块的正确大小的奖金问题；这在很大程度上取决于您的硬件和软件配置。最好的选择是编写一个测试客户端并调制块大小，直到找到服务器的最佳位置。

Part of application I am writing requires transferring arbitrarily big (for this question I will assume 100-200 GB) files from client to server. Important thing, is that receiver (server) is not storing this file - it just read/examine stream and sends it to next point. Because at no point I need whole file, but expect multiple transfers at same time, I would like to minimize RAM usage and eliminate disk usage. I would like to process files in chunks of 1 MB.

Right now, server uses Spring Boot and Akka.

My first attempt was to open buffered file input stream on client side, read it in chunks of 1 MB and send them in messages in separate thread. It works, however problem is that client is sending messages one-after-another without worrying if server has buffer to store it (lacks back pressure).

My second idea was to use akka-streams like this:

How to use Reactive Streams for NIO binary processing?

with use of ActorPublisher like this:

akka-streams with akka-cluster

however, as stated here:

http://doc.akka.io/docs/akka/2.4.16/scala/stream/stream-integrations.html#Implementing_Reactive_Streams_Publisher_or_Subscriber

"Warning ActorPublisher and ActorSubscriber will probably be deprecated in future versions of Akka.

Warning ActorPublisher and ActorSubscriber cannot be used with remote actors, because if signals of the Reactive Streams protocol (e.g. request) are lost the the stream may deadlock."

it doesn't look like good idea.

I do not want to save it in any storage provider (dropbox, google drive, ...) because I want to analyze data on-the-fly. I have Spring 5 and Akka on board, but I can use any other software, that will solve this. Raw socket will lack back pressure and torrents do not guarantee sequential/ordered read-write (that I need).

Main question is: how to stream big file from client to server, assuming server is not able to store file at once on disk nor in ram?

Bonus question is: how to calculate "correct" size of chunk in such transfer?

I have been looking for answer for days, and looks like I am not the only one with problem like this, however there are no answers or answers like "don't do it" without pointing out other adequate alternate solution.

解决方案

Akka stream provides functionality specifically for this use case: streaming File IO. From the documentation:

import akka.stream.scaladsl._
val file = Paths.get("example.csv")

val foreach: Future[IOResult] = 
  FileIO.fromPath(file)
        .to(Sink.ignore)
        .run()

Regarding your bonus question for the "correct size" of chunk; this is highly dependent on your hardware and software configurations. Your best bet is to write a test client and modulate the chunk size until you find a "sweet spot" for your server.

这篇关于通过响应流发送大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！