有没有办法腌制scipy.interpolate.Rbf()对象? [英] Is there a way to pickle a scipy.interpolate.Rbf() object?
问题描述
我正在为一个相当大的数据集创建一个径向基函数插值模型.主要调用`scipy.interpolate.Rbf(，)大约需要一分钟和14 GB的RAM. 由于并非应该在其上运行的每台计算机都能够执行此操作，并且由于该程序会经常在同一数据集上运行，因此我想将结果腌制到一个文件中.这是一个简化的示例:
I'm creating a radial basis function interpolation model for a rather large dataset. The main call `scipy.interpolate.Rbf(,) takes about one minute and 14 GB of RAM. Since not every machine this is supposed to run on is capable of doing this, and since the program will run on the same dataset very often, I'd like to pickle the results to a file. This is a simplified example:
import scipy.interpolate as inter
import numpy as np
import cPickle
x = np.array([[1,2,3],[3,4,5],[7,8,9],[1,5,9]])
y = np.array([1,2,3,4])
rbfi = inter.Rbf(x[:,0], x[:,1], x[:,2], y)
RBFfile = open('picklefile','wb')
RBFpickler = cPickle.Pickler(RBFfile,protocol=2)
RBFpickler.dump(rbfi)
RBFfile.close()
RBFpickler.dump()调用导致can't pickle <type 'instancemethod'>错误.
据我了解，这意味着那里有一个方法(rbfi()是可调用的)，由于某种我不太了解的原因而无法腌制.
The RBFpickler.dump() call results in a can't pickle <type 'instancemethod'> error.
As I understand, that means there's a method somewhere in there (well, rbfi() is callable), and that can't be pickled for some reason I do not quite understand.
有人知道以其他方式腌制还是以其他方式保存inter.Rbf()调用结果的方法吗?
Does anyone know a way of either pickling this in some other way or saving the results of the inter.Rbf() call in some other way?
其中有一些形状为(nd，n)和(n，n)的数组(rbfi.A，rbfi.xi，rbfi.di ...)，我假设它们存储了所有有趣的信息.我想我只能腌制那些数组，但是然后我不确定如何将对象再次放在一起……
There are some arrays of shape (nd,n) and (n,n) in there (rbfi.A, rbfi.xi, rbfi.di...), which I assume store all the interesting information. I guess I could pickle just those arrays, but then I'm not sure how I could put the object together again...
修改: 附加限制:我不允许在系统上安装其他库.我可以包含它们的唯一方法是，如果它们是纯Python，并且可以将它们包含在脚本中，而无需编译任何内容.
Additional constraint: I'm not allowed to install additional libraries on the system. The only way I can include them is if they are pure Python and I can just include them with the script without having to compile anything.
推荐答案
我将使用dill来序列化结果…或者如果您想要具有缓存的函数，则可以使用klepto来缓存函数调用，因此您应尽量减少对该功能的重新评估.
I'd use dill to serialize the results… or if you want to have a cached function you could use klepto to cache the function call so you'd minimize reevaluation of the function.
Python 2.7.6 (default, Nov 12 2013, 13:26:39)
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy.interpolate as inter
>>> import numpy as np
>>> import dill
>>> import klepto
>>>
>>> x = np.array([[1,2,3],[3,4,5],[7,8,9],[1,5,9]])
>>> y = np.array([1,2,3,4])
>>>
>>> # build an on-disk archive for numpy arrays,
>>> # with a dictionary-style interface
>>> p = klepto.archives.dir_archive(serialized=True, fast=True)
>>> # add a caching algorithm, so when threshold is hit,
>>> # memory is dumped to disk
>>> c = klepto.safe.lru_cache(cache=p)
>>> # decorate the target function with the cache
>>> c(inter.Rbf)
<function Rbf at 0x104248668>
>>> rbf = _
>>>
>>> # 'rbf' is now cached, so all repeat calls are looked up
>>> # from disk or memory
>>> d = rbf(x[:,0], x[:,1], x[:,2], y)
>>> d
<scipy.interpolate.rbf.Rbf object at 0x1042454d0>
>>> d.A
array([[ 1. , 1.22905719, 2.36542472, 1.70724365],
[ 1.22905719, 1. , 1.74422655, 1.37605151],
[ 2.36542472, 1.74422655, 1. , 1.70724365],
[ 1.70724365, 1.37605151, 1.70724365, 1. ]])
>>>
继续…
>>> # the cache is serializing the result object behind the scenes
>>> # it also works if we directly pickle and unpickle it
>>> _d = dill.loads(dill.dumps(d))
>>> _d
<scipy.interpolate.rbf.Rbf object at 0x104245510>
>>> _d.A
array([[ 1. , 1.22905719, 2.36542472, 1.70724365],
[ 1.22905719, 1. , 1.74422655, 1.37605151],
[ 2.36542472, 1.74422655, 1. , 1.70724365],
[ 1.70724365, 1.37605151, 1.70724365, 1. ]])
>>>
在此处获取klepto和dill: https://github.com/uqfoundation
这篇关于有没有办法腌制scipy.interpolate.Rbf()对象?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
