Python cPickle:加载失败，出现UnpicklingError [英] Python cPickle: load fails with UnpicklingError

问题描述

我用以下方法制作了一个泡菜文件.

I've made a pickle file using the following.

from PIL import Image
import pickle
import os
import numpy
import time

trainpixels = numpy.empty([80000,6400])
trainlabels = numpy.empty(80000)
validpixels = numpy.empty([10000,6400])
validlabels = numpy.empty(10000)
testpixels = numpy.empty([10408,6400])
testlabels = numpy.empty(10408)

i=0
tr=0
va=0
te=0
for (root, dirs, filenames) in os.walk(indir1):
    print 'hello'
    for f in filenames:
            try:
                    im = Image.open(os.path.join(root,f))
                    Imv=im.load()
                    x,y=im.size
                    pixelv = numpy.empty(6400)
                    ind=0
                    for ii in range(x):
                            for j in range(y):
                                    temp=float(Imv[j,ii])
                                    temp=float(temp/255.0)
                                    pixelv[ind]=temp
                                    ind+=1
                    if i<40000:
                            trainpixels[tr]=pixelv
                            tr+=1
                    elif i<45000:
                            validpixels[va]=pixelv
                            va+=1
                    else:
                            testpixels[te]=pixelv
                            te+=1
                    print str(i)+'\t'+str(f)
                    i+=1
            except IOError:
                    continue
trainimage=(trainpixels,trainlabels)
validimage=(validpixels,validlabels)
testimage=(testpixels,testlabels)

output=open('data.pkl','wb')

pickle.dump(trainimage,output)
pickle.dump(validimage,output)
pickle.dump(testimage,output)

现在，我要解开以下代码的load_data()函数: http://www.deeplearning.net/tutorial/code/logistic_sgd.py 通过运行来调用 http://www.deeplearning.net/tutorial/code/rbm.py

Now I'm unpickling with load_data() function of the following code: http://www.deeplearning.net/tutorial/code/logistic_sgd.py which is called by running http://www.deeplearning.net/tutorial/code/rbm.py

但它返回以下错误.

cPickle.UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.

似乎数据结构是无与伦比的，但我无法弄清楚应该如何.

It seems like data structure is unmatched, but I can' figure out how it should be..

作为参考，泡菜文件的大小超过16GB，其gzip超过1GB

For reference, the size of the pickle file is over 16GB, with its gzip over 1GB

推荐答案

我发现酸洗和酸洗很聪明. 在这里，您不会像腌制一样去腌制，因此它无法工作.在您的代码中，您在同一文件中一个接一个地腌制对象.您腌制了3次至同一文件. 如果您想读回它们，则必须进行顺序阅读. 您需要做的是打开文件以进行解腌，然后依次pickle.load每个对象.

I've found that pickling and unpickling is smart. Here you don't unpickle the same way you pickle, so it cannot work. In your code you pickle objects one after the other in the same file. You pickled three times to the same file. If you want to read them back, you have to make sequential reading. What you have to do is open the file for unpickling, then pickle.load each of your objects sequentially.

with gzip.open(dataset, 'rb') as f:
    train_set = cPickle.load(f)
    valid_set = cPickle.load(f)
    test_set = cPickle.load(f)

您可能想要尝试一个更简单的代码，其中train_set, valid_set, test_set(使用gzip进行酸洗和酸洗)是简单的可酸洗对象，只是为了确定.

You might want to try a simpler code where train_set, valid_set, test_set (do the pickling and unpickling with gzip) are simple picklable objects, just to be sure.

这篇关于Python cPickle:加载失败，出现UnpicklingError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！