Cython:类型化的memoryviews是键入numpy数组的现代方法吗? [英] Cython: are typed memoryviews the modern way to type numpy arrays?

问题描述

假设我想将numpy数组传递给cdef函数:

Let's say I'd like to pass a numpy array to a cdef function:

cdef double mysum(double[:] arr):
    cdef int n = len(arr)
    cdef double result = 0

    for i in range(n):
        result = result + arr[i]

    return result

这是处理键入numpy数组的现代方法吗?与此问题进行比较: cython/数组的numpy类型

Is this the modern way to handle typing numpy arrays? Compare with this question: cython / numpy type of an array

如果我想执行以下操作怎么办?

What if I want to do the following:

cdef double[:] mydifference(int a, int b):
    cdef double[:] arr_a = np.arange(a)
    cdef double[:] arr_b = np.arange(b)

    return arr_a - arr_b

这将返回错误，因为未为memoryviews定义-.那么，该案件应按以下方式处理吗?

This will return an error because - is not defined for memoryviews. So, should that case have been handled as follows?

cdef double[:] mydifference(int a, int b):
    arr_a = np.arange(a)
    arr_b = np.arange(b)

    return arr_a - arr_b

推荐答案

我将引用文档文档

内存视图类似于当前的NumPy数组缓冲区支持(np.ndarray[np.float64_t, ndim=2])，但是它们具有更多功能和更简洁的语法.

Memoryviews are similar to the current NumPy array buffer support (np.ndarray[np.float64_t, ndim=2]), but they have more features and cleaner syntax.

这表明Cython的开发人员认为内存视图是现代方法.

This indicates that the developers of Cython consider memory views to be the modern way.

内存视图相对于np.ndarray表示法具有一些重大优势，主要是在美观和互操作性方面，但在性能上并不优越.

Memory views offer some big advantages over the np.ndarray notation primarily in elegance and interoperability, however they are not superior in performance.

首先应该注意，boundscheck 有时不能与内存视图配合使用，如果您依赖的话，会导致为boundscheck = True的内存视图提供人为的快速数字(即，获得快速，不安全的索引)在boundscheck上捕获错误，这可能是一个令人讨厌的惊喜.

First it should be noted that boundscheck sometimes fails to work with memory views resulting in artificially fast figures for memoryviews with boundscheck=True (i.e. you get fast, unsafe indexing), if you're relying on boundscheck to catch bugs this could be a nasty surprise.

在大多数情况下，一旦应用了编译器优化，内存视图和numpy数组表示法的性能就相同，通常也是如此.当存在差异时，通常不超过10-30％.

For the most part once compiler optimizations have been applied, memory views and numpy array notation are equal in performance, often precisely so. When there is a difference it is normally no more than 10-30%.

数字是执行100,000,000次操作的时间(以秒为单位).越小越快.

The number is the time in seconds to perform 100,000,000 operations. Smaller is faster.

ACCESS+ASSIGNMENT on small array (10000 elements, 10000 times)
Results for `uint8`
1) memory view: 0.0415 +/- 0.0017
2) np.ndarray : 0.0531 +/- 0.0012
3) pointer    : 0.0333 +/- 0.0017

Results for `uint16`
1) memory view: 0.0479 +/- 0.0032
2) np.ndarray : 0.0480 +/- 0.0034
3) pointer    : 0.0329 +/- 0.0008

Results for `uint32`
1) memory view: 0.0499 +/- 0.0021
2) np.ndarray : 0.0413 +/- 0.0005
3) pointer    : 0.0332 +/- 0.0010

Results for `uint64`
1) memory view: 0.0489 +/- 0.0019
2) np.ndarray : 0.0417 +/- 0.0010
3) pointer    : 0.0353 +/- 0.0017

Results for `float32`
1) memory view: 0.0398 +/- 0.0027
2) np.ndarray : 0.0418 +/- 0.0019
3) pointer    : 0.0330 +/- 0.0006

Results for `float64`
1) memory view: 0.0439 +/- 0.0037
2) np.ndarray : 0.0422 +/- 0.0013
3) pointer    : 0.0353 +/- 0.0013

ACCESS PERFORMANCE (100,000,000 element array):
Results for `uint8`
1) memory view: 0.0576 +/- 0.0006
2) np.ndarray : 0.0570 +/- 0.0009
3) pointer    : 0.0061 +/- 0.0004

Results for `uint16`
1) memory view: 0.0806 +/- 0.0002
2) np.ndarray : 0.0882 +/- 0.0005
3) pointer    : 0.0121 +/- 0.0003

Results for `uint32`
1) memory view: 0.0572 +/- 0.0016
2) np.ndarray : 0.0571 +/- 0.0021
3) pointer    : 0.0248 +/- 0.0008

Results for `uint64`
1) memory view: 0.0618 +/- 0.0007
2) np.ndarray : 0.0621 +/- 0.0014
3) pointer    : 0.0481 +/- 0.0006

Results for `float32`
1) memory view: 0.0945 +/- 0.0013
2) np.ndarray : 0.0947 +/- 0.0018
3) pointer    : 0.0942 +/- 0.0020

Results for `float64`
1) memory view: 0.0981 +/- 0.0026
2) np.ndarray : 0.0982 +/- 0.0026
3) pointer    : 0.0968 +/- 0.0016

ASSIGNMENT PERFORMANCE (100,000,000 element array):
Results for `uint8`
1) memory view: 0.0341 +/- 0.0010
2) np.ndarray : 0.0476 +/- 0.0007
3) pointer    : 0.0402 +/- 0.0001

Results for `uint16`
1) memory view: 0.0368 +/- 0.0020
2) np.ndarray : 0.0368 +/- 0.0019
3) pointer    : 0.0279 +/- 0.0009

Results for `uint32`
1) memory view: 0.0429 +/- 0.0022
2) np.ndarray : 0.0427 +/- 0.0005
3) pointer    : 0.0418 +/- 0.0007

Results for `uint64`
1) memory view: 0.0833 +/- 0.0004
2) np.ndarray : 0.0835 +/- 0.0011
3) pointer    : 0.0832 +/- 0.0003

Results for `float32`
1) memory view: 0.0648 +/- 0.0061
2) np.ndarray : 0.0644 +/- 0.0044
3) pointer    : 0.0639 +/- 0.0005

Results for `float64`
1) memory view: 0.0854 +/- 0.0056
2) np.ndarray : 0.0849 +/- 0.0043
3) pointer    : 0.0847 +/- 0.0056

基准代码(仅针对访问和分配显示)

# cython: boundscheck=False
# cython: wraparound=False
# cython: nonecheck=False
import numpy as np
cimport numpy as np
cimport cython

# Change these as desired.
data_type = np.uint64
ctypedef np.uint64_t data_type_t

cpdef test_memory_view(data_type_t [:] view):
    cdef Py_ssize_t i, j, n = view.shape[0]

    for j in range(0, n):
        for i in range(0, n):
            view[i] = view[j]

cpdef test_ndarray(np.ndarray[data_type_t, ndim=1] view):
    cdef Py_ssize_t i, j, n = view.shape[0]

    for j in range(0, n):
        for i in range(0, n):
            view[i] = view[j]

cpdef test_pointer(data_type_t [:] view):
    cdef Py_ssize_t i, j, n = view.shape[0]
    cdef data_type_t * data_ptr = &view[0]

    for j in range(0, n):
        for i in range(0, n):
            (data_ptr + i)[0] = (data_ptr + j)[0]

def run_test():
    import time
    from statistics import stdev, mean
    n = 10000
    repeats = 100
    a = np.arange(0, n,  dtype=data_type)
    funcs = [('1) memory view', test_memory_view),
        ('2) np.ndarray', test_ndarray),
        ('3) pointer', test_pointer)]

    results = {label: [] for label, func in funcs}
    for r in range(0, repeats):
        for label, func in funcs:
            start=time.time()
            func(a)
            results[label].append(time.time() - start)

    print('Results for `{}`'.format(data_type.__name__))
    for label, times in sorted(results.items()):
        print('{: <14}: {:.4f} +/- {:.4f}'.format(label, mean(times), stdev(times)))

这些基准表明，总体而言，性能没有太大差异.有时np.ndarray表示法要快一些，有时反之亦然.

These benchmarks indicate that on the whole there is not much difference in performance. Sometimes the np.ndarray notation is a little faster, and sometimes vice-verca.

基准测试需要注意的一件事是，当代码变得更加复杂或逼真的"时，差异突然消失，好像编译器失去了应用某些非常聪明的优化的信心.可以从浮点数的性能中看出这一点，因为可能无法使用某些奇特的整数优化，所以两者之间没有任何区别.

One thing to watch out for with benchmarks is that when the code is made a little bit more complicated or 'realistic' the difference suddenly vanishes, as if the compiler loses confidence to apply some very clever optimization. This can be seen with the performance of floats where there is no difference whatsoever presumably as some fancy integer optimizations can't be used.

内存视图提供了显着的优势，例如，您可以在numpy数组，CPython数组，cython数组，c数组等上使用内存视图，无论现在还是将来.还有一种简单的并行语法可以将任何内容强制转换为内存视图:

Memory views offer significant advantages, for example you can use a memory view on numpy array, CPython array, cython array, c array and more, both present and future. There is also the simple parallel syntax for casting anything to a memory view:

cdef double [:, :] data_view = <double[:256, :256]>data

在这方面内存视图非常有用，因为如果您键入一个函数作为内存视图，那么它可以接任何这些东西.这意味着您可以编写一个不依赖numpy的模块，但是仍然可以使用numpy数组.

Memory views are great in this regard, because if you type a function as taking a memory view then it can take any of those things. This means you can write a module that doesn't have a dependency on numpy, but which can still take numpy arrays.

另一方面，np.ndarray表示法产生的结果仍然是一个numpy数组，您可以在其上调用所有numpy数组方法.虽然同时拥有一个numpy数组和对该数组的视图并不重要:

On the other hand, np.ndarray notation results in something that is still a numpy array and you can call all the numpy array methods on it. It's not a big deal to have both a numpy array and a view on the array though:

def dostuff(arr):
    cdef double [:] arr_view = arr
    # Now you can use 'arr' if you want array functions,
    # and arr_view if you want fast indexing

在实践中同时拥有数组和数组视图都很好，我非常喜欢这种样式，因为它在python级方法和c级方法之间有明显的区别.

Having both the array and the array view works fine in practise and I quite like the style, as it makes a clear distinction between python-level methods and c-level methods.

性能几乎是相等的，并且肯定没有足够的差异作为决定因素.

Performance is very nearly equal and there is certainly not enough difference for that to be a deciding factor.

numpy数组符号更接近于在不进行太多更改的情况下加速python代码的理想状态，因为您可以继续使用相同的变量，同时获得全速数组索引.

The numpy array notation comes closer to the ideal of accelerating python code without changing it much, as you can continue to use the same variable, while gaining full-speed array indexing.

另一方面，内存视图符号可能是未来.如果您喜欢它的优雅，并且使用了除numpy数组以外的其他类型的数据容器，那么出于一致性的考虑，有充分的理由使用内存视图.

On the other hand, the memory view notation probably is the future. If you like the elegance of it, and use different kinds of data containers than just numpy arrays, there is very good reason for using memory views for consistency's sake.

这篇关于Cython:类型化的memoryviews是键入numpy数组的现代方法吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！