通过将rbf与scipy一起使用会导致内存错误 [英] memory error by using rbf with scipy

问题描述

我想使用rbf函数绘制一些点，如下所示，以获得点的密度分布:

I want to plot some points with the rbf function like here to get the density distribution of the points:

如果我运行以下代码，则可以正常运行:

if i run the following code, it works fine:

from scipy.interpolate.rbf import Rbf  # radial basis functions
import cv2
import matplotlib.pyplot as plt
import numpy as np

# import data
x = [1, 1, 2 ,3, 2, 7, 8, 6, 6, 7, 6.5, 7.5, 9, 8, 9, 8.5]
y = [0, 2, 5, 6, 1, 2, 9, 2, 3, 3, 2.5, 2, 8, 8, 9, 8.5]
d = np.ones(len(x))
print(d)

ti = np.linspace(-1,10)

xx, yy = np.meshgrid(ti, ti)

rbf = Rbf(x, y, d, function='gaussian')
jet = cm = plt.get_cmap('jet')
zz = rbf(xx, yy)
plt.pcolor(xx, yy, zz, cmap=jet)
plt.colorbar()

# Plotting the original points.
plot3 = plt.plot(x, y, 'ko', markersize=5)  # the original points.

plt.show()

现在我要更改输入数据:

Now I want to change my input data:

# import data
input = "testProbe.jpg"
image = cv2.imread(input)   # load the image
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)  # convert it to grayscale
# threshold the image to reveal light regions in the gray image
thresh = cv2.threshold(gray, 145, 200, cv2.THRESH_BINARY)[1]

x, y = np.where(thresh > 0)
d = np.ones(len(x))

我收到以下错误消息:

Traceback (most recent call last):
  File "C:/Users/.../Python/pythonprojects/03a_test_rbfWithScipy.py", line 32, in <module>
    rbf = Rbf(x, y, z, function='gaussian')
  File "C:\Users\...\Python\Anaconda2\lib\site-packages\scipy\interpolate\rbf.py", line 220, in __init__
    self.nodes = linalg.solve(self.A, self.di)
  File "C:\Users\...\Python\Anaconda2\lib\site-packages\scipy\interpolate\rbf.py", line 226, in A
    r = self._call_norm(self.xi, self.xi)
  File "C:\Users\...\Python\Anaconda2\lib\site-packages\scipy\interpolate\rbf.py", line 236, in _call_norm
    return self.norm(x1, x2)
  File "C:\Users\...\Python\Anaconda2\lib\site-packages\scipy\interpolate\rbf.py", line 118, in _euclidean_norm
    return np.sqrt(((x1 - x2)**2).sum(axis=0))
MemoryError

错误消息的显示速度非常快，当我查看任务管理器时，PC并未满负荷运行.他立即发出此消息.那时有些不对，对吗?

The error message appears really fast and when I look at the task manager, the PC is not running at full capacity. He issues this message immediately. Something can't be right then, can't it?

我尝试使用其他阈值thresh = cv2.threshold(gray, 250, 255, cv2.THRESH_BINARY)[1]来获得更少的值，尽管我收到以下错误消息，它仍然有效:

I tried it with other threshold values thresh = cv2.threshold(gray, 250, 255, cv2.THRESH_BINARY)[1] to get less values and it works although I get the following error message:

C:\Users\...\Python\Anaconda2\lib\site-packages\scipy\interpolate\rbf.py:220: LinAlgWarning: 
scipy.linalg.solve Ill-conditioned matrix detected. Result is not guaranteed to be accurate. 
Reciprocal condition number2.246772e-22 self.nodes = linalg.solve(self.A, self.di)

有什么想法吗?

如果我只剪掉图片的一小部分，那么它也可以使用.也许它对内存进行了一些初步计算，并给了我内存错误

If I only cut out a small part of the picture, then it works too. Maybe it makes some preliminary calculation for the memory and gives me the memory error

推荐答案

在我看来，Rbf插值计算量很大，并且会导致O(N^2)运算(如果我错了，请纠正我).因此，为了避免出现内存错误，您可以执行以下操作之一，而不是

It appears to me Rbf interpolation is quite computationally intensive and leads to O(N^2) operations (correct me if I am wrong). So in order to avoid memory error you can do one of the following, instead of

zz = rbf(xx, yy)

1.用nditer迭代

速度慢，但适用于小型阵列:

Slow, but works for small arrays:

for iz, ix, iy in np.nditer(
    [zz, xx, yy],
    flags=['external_loop', 'buffered'], 
    op_flags=['readwrite']
):
    iz[...] = rbf(ix, iy)

2.使用dask.array

更快的选项并使用线程

import dask.array as da

n1 = xx.shape[1]
ix = da.from_array(xx, chunks=(1, n1))
iy = da.from_array(yy, chunks=(1, n1))
iz = da.map_blocks(rbf, ix, iy)
zz = iz.compute()

希望这对您有用.

PS :如另一个问题和 LowLevelCallable 和 Pythran博客).它涉及到一点点，但它也可以解决您的问题，同时还可以提高性能.

P.S: As suggested on another question and scipy docs on Rbf, Rbf also allows you to replace the norm calculation function with a callable (see LowLevelCallable and the Pythran blog). It is a bit involved, but it might also solve your problem, while increasing the performance as well.

P.P.S :通过进行以下修改，我设法将速度提高了50倍:

P.P.S: I managed to speedup by a factor of 50 by making this modification:

import numpy as np

def euclidean_norm_numpy(x1, x2):
    return np.linalg.norm(x1 - x2, axis=0)

rbf = Rbf(x, y, d, function='gaussian', norm=euclidean_norm_numpy)

这篇关于通过将rbf与scipy一起使用会导致内存错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！