商品Linux存储农场的最佳分布式文件系统 [英] Best distributed filesystem for commodity linux storage farm

问题描述

我有大量闲置的intel linux服务器（数百个），希望将它们用于Web托管和文件共享环境中的分布式文件系统。这不适用于HPC应用程序，因此高性能并不重要。主要的要求是高可用性，如果一台服务器脱机，存储在硬盘上的数据仍然可以从其他节点获得。它必须运行在TCP / IP上，并提供标准的POSIX文件许可权。

我看了下面的内容：

• Lustre（ http：//wiki.lustre .org / index.php？title = Main_Page ）：真的关闭，但它不提供节点上的数据冗余。您必须使用RAID或DRBD制作数据HA。由Sun和Open Source支持，所以它应该在一段时间内

gfarm（ http://datafarm.apgrid.org/ ）：看起来它提供了冗余，但代价是复杂性和可维护性。没有Lustre的支持。

有没有人有这些或任何其他系统的经验？ / p>

解决方案

也请 GlusterFS

编辑（2012年8月） Ceph 终于准备好了。最近作者成立了一家独立的公司，出售 Inktank 。根据一些介绍，可挂载的POSIX兼容的文件系统是最上面的一层，并没有真正的测试，但是较低层现在正在使用中。

有趣的部分是RADOS层，它通过 librados 库（可用于多种语言）和Amazon S3兼容的RESP API。任何一个都不足以将大量存储添加到Web服务。

这个视频是对哲学，架构，功能和当前状态的一个很好的描述。

I have a lot of spare intel linux servers laying around (hundreds) and want to use them for a distributed file system in a web hosting and file sharing environment. This isn't for a HPC application, so high performance isn't critical. The main requirement is high availability, if one server goes offline, the data stored on it's hard drives is still available from other nodes. It must run over TCP/IP and provide standard POSIX file permissions.

I've looked at the following:

• Lustre (http://wiki.lustre.org/index.php?title=Main_Page): Comes really close, but it doesn't provide redundancy for data on a node. You must make the data HA using RAID or DRBD. Supported by Sun and Open Source, so it should be around for a while

• gfarm (http://datafarm.apgrid.org/): Looks like it provides the redundancy but at the cost of complexity and maintainability. Not as well supported as Lustre.

Does anyone have any experience with these or any other systems that might work?

解决方案

check also GlusterFS

Edit (Aug-2012): Ceph is finally getting ready. Recently the authors formed Inktank, an independent company to sell commercial support for it. According to some presentaions, the mountable POSIX-compliant filesystem is the uppermost layer and not really tested yet, but the lower layers are being used in production for some time now.

The interesting part is the RADOS layer, which presents an object-based storage with both a 'native' access via the librados library (available for several languages) and an Amazon S3-compatible RESP API. Either one makes it more than adequate for adding massive storage to a web service.

This video is a good description of the philosophy, architecture, capabilities and current status.

这篇关于商品Linux存储农场的最佳分布式文件系统的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！