快速访问多个VM上的数据 [英] Fast access to data on multiple VMs

问题描述

大家好。

我正在寻找一种解决方案来存储来自多个虚拟机（主要是基于Linux的）集中式数据。有些虚拟机可以生成数据（运行模拟），还有用于分析数据的虚拟机。每个VM都专门针对特定目标。

生成数据的VM需要快速写访问权，而用于分析数据的VM需要快速读取访问权限。每个作业的数据大小是几百GB。

如果我直接将数据写入blob，写入速度非常低，这将导致瓶颈，并且模拟运行时间会急剧增加。此外，对数据的访问将是缓慢的，分析将是一个痛苦。但我有一个集中的地方
我的所有数据。

如果我将数据写入VM上的本地磁盘，我需要以某种方式将数据传输到另一个VM到分析它。我可以分离磁盘并连接到另一个VM，但我希望能够在模拟仍在运行时分析数据。

我可以想象，将数据写入本地磁盘，将其复制到blob，然后将其从blob复制到另一个VM的本地磁盘将起作用。这听起来像是一个合理的想法吗？我可以使用azcopy，并且足够快，可以在几分钟内复制数百美元b $ b GB吗？

有没有人有类似情况的经验，你是如何解决的？

提前致谢并度过愉快的一天。

解决方案

< blockquote>

您是否考虑在同一虚拟网络上配置运行Samba和高级存储的Windows文件服务器或Linux服务器以提供通用数据存储？

hth

Marcin

Hello everybody

I'm looking for a solution to store data from multiple VMs (mostly Linux based) centralized. There are VMs that produce the data (running simulations) and there are VMs which are used to analyze the data. Each VM is specialized for the specific taks.

The VMs that produce the data needs fast write access and the VMs for analyzing the data need fast read access. The data size per job is several hundred GBs.

If i write data to a blob directly, the write speed is quite low and this will result in a bottleneck and the simulation runtime will increase drastically. Also the access to the data will be slow and analyzing will be a pain. But i have a centralized place for all my data.

If i write data to a local disk on the VM, I need somehow to transfer that data to the other VM to analyze it. I could detach the disk and attach to the other VM but i want to be able to analyze the data while the simulation is still running.

I could imagine, that writing data to a local disk, copy it to a blob and then copy it from the blob to a local disk of the other VM would work. Does that sound like a reasonable idea? Can I use azcopy for that and is that fast enough to copy several hundred GBs in few minutes?

Does anyone have experience with a similar situation and how did you solve that?

Thanks in advance and have a nice day.

解决方案

Have you considered provisioning a Windows file server or a Linux server running Samba with Premium storage on the same virtual network to provide common data store?

hth
Marcin

这篇关于快速访问多个VM上的数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！