Python:存储大数据结构 [英] Python: storing big data structures

问题描述

我目前正在用python做一个项目，该项目使用相对较大的字典(大约800 MB).我试图通过使用泡菜来存储其中一个词典，但是遇到了MemoryError.

I'm currently doing a project in python that uses dictionaries that are relatively big (around 800 MB). I tried to store one of this dictionaries by using pickle, but got an MemoryError.

在python中保存此类文件的正确方法是什么?我应该使用数据库吗?

What is the proper way to save this kind of files in python? Should I use a database?

推荐答案

Python标准保护模块为持久对象提供类似于dict的界面.它适用于许多数据库后端，不受RAM的限制.使用shelve而不是直接使用数据库的优点是，大多数现有代码保持原样.这样做的代价是速度(与RAM中的指令相比)和灵活性(与直接与数据库一起使用).

Python-standard shelve module provides dict-like interface for persistent objects. It works with many database backends and is not limited by RAM. The advantage of using shelve over direct work with databases is that most of your existing code remains as it was. This comes at the cost of speed (compared to in-RAM dicts) and at the cost of flexibility (compared to working directly with databases).

这篇关于Python:存储大数据结构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！