比较python中的持久存储解决方案 [英] Comparing persistent storage solutions in python

问题描述

我正在开始一个新的科学项目，该项目包含大量数据(数百万个条目)，我想以一种易于快速访问的格式进行存储.我遇到了许多不同的潜在选择，但是我不确定如何选择它们.我的数据可能只是存储为字典，也可能是字典的字典.一些潜在的注意事项:

I'm starting on a new scientific project which has a lot of data (millions of entries) I'd like to store in an easily and quickly accessible format. I've come across a number of different potential options, but I'm not sure how to pick amongst them. My data can probably just be stored as a dictionary, or potentially a dictionary of dictionaries. Some potential considerations:

• 速度.我每次启动新脚本时都无法从磁盘上加载所有数据，并且希望尽快访问随机条目.
• 易于使用.这是python.存储应该感觉像python.
• 稳定性/成熟度.我想要一些目前受支持的东西，尽管效果很好但仍在开发中就可以了.
• 易于安装.我的系统管理员应该能够在集群上运行它.

我不太在乎存储的大小，但是如果这方面的选择真的很糟糕，则可以考虑.另外，如果有关系，我很可能会创建一次数据库，然后仅从数据库中读取.

I don't really care that much about the size of the storage, but it could be a consideration if an option is really terrible on this front. Also, if it matters, I'll most likely be creating the database once, and thereafter only reading from it.

我已经开始研究的一些潜在选项(请参阅

Some potential options that I've started looking at (see this post):

• pyTables
• ZopeDB
• 推
• 货架
• redis
• durus

• pyTables
• ZopeDB
• shove
• shelve
• redis
• durus

对于我的目的而言，关于其中哪一项可能更好的任何建议?还有更好的主意吗?其中一些具有后端.关于哪种文件系统后端最好的任何建议?

Any suggestions on which of these might be better for my purposes? Any better ideas? Some of these have a back-end; any suggestions on which file-system back-end would be best?

推荐答案

RDBMS.

没有什么比在众所周知的RDBMS上使用表更现实的了.我想到了 PostgreSQL .

Nothing is more realiable than using tables on a well known RDBMS. Postgresql comes to mind.

这会自动为您提供一些将来的选择，例如集群.另外，您会自动拥有许多用于管理数据库的工具，并且可以从实际上以任何语言编写的其他软件中使用它.

That automatically gives you some choices for the future like clustering. Also you automatically have a lot of tools to administer your database, and you can use it from other software written in virtually any language.

这真的很快.

在感觉像python"方面，我可能会补充说您可以使用ORM.一个强名是 sqlalchemy .也许使用 elixir "扩展".

In the "feel like python" point, I might add that you can use an ORM. A strong name is sqlalchemy. Maybe with the elixir "extension".

使用sqlalchemy，您可以让用户/系统管理员选择他要使用的数据库后端.也许他们已经安装了 MySql -没问题.

Using sqlalchemy you can leave your user/sysadmin choose which database backend he wants to use. Maybe they already have MySql installed - no problem.

RDBMS仍然是数据存储的最佳选择.

RDBMSs are still the best choice for data storage.

这篇关于比较python中的持久存储解决方案的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！