Pickle与Shelve在Python中存储大型词典 [英] Pickle versus shelve storing large dictionaries in Python

问题描述

如果我将大目录存储为pickle文件，通过cPickle加载是否意味着它会立即全部消耗到内存中?

If I am storing a large directory as a pickle file, does loading it via cPickle mean that it will all be consumed into memory at once?

如果是这样，是否有一种跨平台的方式来获取类似pickle的内容，但是在每个项上访问一个键(即避免将所有字典加载到内存中，而是仅按名称加载每个项)?我知道shelve应该这样做:像pickle一样可移植吗?

If so, is there a cross platform way to get something like pickle, but access each entry one key at a item (i.e. avoid loading all of the dictionary into memory and only load each entry by name)? I know shelve is supposed to do this: is that as portable as pickle though?

推荐答案

我知道应该货架上做这个:像泡菜一样轻巧吗?

I know shelve is supposed to do this: is that as portable as pickle though?

是的. shelve 是

Yes. shelve is part of The Python Standard Library and is written in Python.

因此，如果您有大型词典:

So if you have a large dictionary:

bigd = {'a': 1, 'b':2, # . . .
}

您想保存它而不必稍后再阅读整个内容，而不必将其保存为泡菜，最好将其保存为一个架子，某种形式的磁盘字典.

And you want to save it without having to read the whole thing in later then don't save it as a pickle, it would be better to save it as a shelf, a sort of on disk dictionary.

import shelve

myShelve = shelve.open('my.shelve')
myShelve.update(bigd)
myShelve.close()

然后您可以:

import shelve

myShelve = shelve.open('my.shelve')
value = myShelve['a']
value += 1
myShelve['a'] = value

您基本上将搁置的对象像字典一样对待，但是将项目存储在磁盘上(作为单独的泡菜)，并根据需要读取.

You basically treat the shelve object like a dict, but the items are stored on disk (as individual pickles) and read in as needed.

如果您的对象可以存储为属性列表，则 sqlite 可能是一个很好的选择.架子和泡菜很方便，但是只能通过Python访问，但是sqlite数据库可以从大多数语言中读取.

If your objects could be stored as a list of properties, then sqlite may be a good alternative. Shelves and pickles are convenient, but can only be accessed by Python, but a sqlite database can by read from most languages.

这篇关于Pickle与Shelve在Python中存储大型词典的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！