NumPy矩阵类的弃用状态 [英] Deprecation status of the NumPy matrix class

问题描述

NumPy中matrix类的状态是什么?

我一直被告知我应该改用ndarray类.在我编写的新代码中使用matrix类是否值得/安全?我不明白为什么我应该改用ndarray s.

解决方案

tl; dr: numpy.matrix类已被弃用.有一些引人注目的库将类作为依赖项(最大的库是scipy.sparse)，这阻碍了该类的短期正确使用，但是强烈建议用户使用ndarray类(通常创建)而是使用 numpy.array 便捷功能).引入@运算符进行矩阵乘法运算后，许多相对的优势都被消除了.

为什么(不是)矩阵类?

numpy.matrix是numpy.ndarray的子类.它最初是为了方便在涉及线性代数的计算中使用，但与更通用的数组类的实例相比，它们在行为方式上既有局限性，又有令人惊讶的差异.行为上的根本差异的示例:

• 形状:数组可以具有任意数量的维，范围从0到无穷大(或32).矩阵始终是二维的.奇怪的是，虽然不能创建具有更大尺寸的矩阵，但也可以将单例尺寸注入到矩阵中，从而在技术上以多维矩阵结尾:np.matrix(np.random.rand(2,3))[None,...,None].shape == (1,2,3,1)(并非这是任意的实际重要性).
• 建立索引:建立索引的数组可以为您提供任意大小的数组，具体取决于如何您正在为其编入索引.在矩阵上建立索引表达式将始终为您提供矩阵.这意味着2d数组的arr[:,0]和arr[0,:]都为您提供1d ndarray，而对于matrix，mat[:,0]具有形状(N,1)，而mat[0,:]具有形状(1,M). /li>
• 算术运算:过去使用矩阵的主要原因是矩阵的算术运算(尤其是乘法和幂)执行矩阵运算(矩阵乘法和矩阵幂).数组的相同结果会导致元素乘法和幂运算.因此，如果mat1.shape[1] == mat2.shape[0]，mat1 * mat2有效，但是如果arr1.shape == arr2.shape，arr1 * arr2有效(当然，结果意味着完全不同).同样，令人惊讶的是，mat1 / mat2对两个矩阵执行 elementwise 除法.此行为可能是从ndarray继承的，但对于矩阵没有意义，尤其是考虑到*的含义.
• 特殊属性:矩阵具有一些方便的属性除了数组具有的内容:mat.A和mat.A1是分别具有与np.array(mat)和np.array(mat).ravel()相同值的数组视图. mat.T和mat.H是矩阵的转置和共轭转置(伴随)； arr.T是ndarray类存在的唯一此类属性.最后，mat.I是mat的逆矩阵.

编写适用于ndarray或矩阵的代码非常容易.但是，当这两个类有可能必须在代码中进行交互时，事情就会变得困难起来.特别是，许多代码可以自然地对ndarray的子类起作用，但是matrix是行为不佳的子类，很容易破坏试图依赖鸭子类型的代码.考虑以下使用形状为(3,4)的数组和矩阵的示例:

import numpy as np

shape = (3, 4)
arr = np.arange(np.prod(shape)).reshape(shape) # ndarray
mat = np.matrix(arr) # same data in a matrix
print((arr + mat).shape)           # (3, 4), makes sense
print((arr[0,:] + mat[0,:]).shape) # (1, 4), makes sense
print((arr[:,0] + mat[:,0]).shape) # (3, 3), surprising

两个对象的切片相加的方式在灾难性方面有所不同，具体取决于我们沿切片的尺寸.当形状相同时，在矩阵和数组上的加法都是按元素进行的.上面的前两种情况很直观:我们添加了两个数组(矩阵)，然后分别添加了两行.最后一种情况确实令人惊讶:我们可能打算添加两列，最后得到一个矩阵.当然，原因是arr[:,0]具有与(1,3)形状兼容的形状(3,)，而mat[:.0]具有(3,1)形状.两者都是广播一起形成(3,3).

最后，当(v_row * mat * v_row.T) * v_row.T会引发错误，因为形状为(1,1)和(3,1)的矩阵不能以此顺序相乘.

为完整性起见，应注意，虽然matmul运算符修复了ndarray与矩阵相比次优的最常见情况，但使用ndarray优雅地处理线性代数仍存在一些缺点(尽管人们仍然相信总体而言，最好坚持使用后者).一个这样的例子是矩阵幂:mat ** 3是矩阵的适当第三矩阵幂(而它是ndarray的元素级立方).不幸的是，numpy.linalg.matrix_power更为冗长.此外，就地矩阵乘法仅适用于矩阵类.相反，虽然 PEP 465 和 同时使用numpy.matrix语义，并且在进行密化处理时经常返回矩阵)，因此完全弃用它们总是有问题的.

已经出现在 2009年的numpy邮件列表线程中我发现了诸如

numpy是为满足通用计算需求而设计的，而不是任何一项 数学的分支. nd数组在很多事情上都非常有用.在 相比之下，例如Matlab最初设计得很简单 线性代数软件包的前端.就个人而言，当我使用Matlab时， 发现那很尴尬-我通常写几百行代码 与线性代数无关，每隔几行 实际上做了矩阵数学.所以我更喜欢numpy的方式-线性 代数代码行更长一些，但是其余的很多 更好.

Matrix类是该类的例外:编写该类是为了提供一个 表达线性代数的自然方法.但是，事情变得有些棘手 当您混合矩阵和数组，甚至坚持矩阵时 有困惑和局限性-您如何表达一行与 列向量?在矩阵上迭代时会得到什么?等

关于这些问题的讨论很多，很多很好 想法，关于如何改进它的一点共识，但是没有人 熟练地做到这一点的人就有足够的动力去做.

这些反映了矩阵类带来的好处和困难.我可以找到的最早的弃用建议是 2008年起 ，尽管部分原因是由于此后发生的非直觉行为所致(特别是对矩阵进行切片和迭代将产生(行)矩阵，这是人们最有可能期望的结果).该建议表明，这是一个备受争议的主题，而且矩阵乘法的中缀运算符至关重要.

我可以找到的下一个提及来自2014年证明是一个非常富有成果的线程.随后的讨论提出了一般处理numpy子类的问题，哪个主题仍然摆在桌子上.还有强烈批评:

引发此讨论(在Github上)的是，不可能 编写适用于以下情况的鸭子式代码:

• ndarrays
• 矩阵
• scipy.sparse稀疏矩阵

这三个词的语义是不同的. scipy.sparse在某处 在矩阵和ndarrays之间，有些东西像随机工作 矩阵而其他则没有.

添加一些双引号，可以从开发人员的角度说 看来，np.matrix正在做，并且已经存在，并且已经做了恶， 通过弄乱Python中ndarray语义的未声明规则.

之后，对矩阵的可能未来进行了许多有价值的讨论.即使当时没有@运算符，人们也对矩阵类的弃用及其对下游用户的影响有很多想法.据我所知，这种讨论直接导致了引入PEmul的PEP 465的诞生.

在2015年初:

我认为，np.matrix的固定"版本不应该(1)是 np.ndarray子类和(2)存在于第三方库中，而不是numpy本身.

我认为将np.matrix修复为当前状态确实不可行 ndarray子类，但即使固定矩阵类也不真正属于 numpy本身，发布周期太长且不兼容 保证实验-更不用说仅仅存在 numpy中的矩阵类使新用户误入歧途.

@运算符一会儿可用关于弃用的讨论再次浮出水面，重新讨论主题关于矩阵弃用与scipy.sparse的关系.

最终，采取了第一个弃用numpy.matrix的措施2017年11月下旬.关于班级的家属:

社区将如何处理scipy.sparse矩阵子类?这些 仍然很常用.

它们很长时间都不会去任何地方了(直到稀疏的ndarrays 至少实现).因此，np.matrix需要移动而不是删除.

(源)和

我想尽可能多地摆脱np.matrix 任何人，很快就会做到这一点真的具有破坏性.

• 那里有很多人写的小脚本 没有更好的了解；我们确实希望他们学习不使用np.matrix而是 破坏所有脚本是一种痛苦的方式

• 像scikit-learn这样的大型项目根本就没有 由于scipy.sparse，可以替代使用np.matrix.

所以我认为前进的方向是这样的:

• 现在或每当有人聚在一起PR时:发出一个 np.matrix .__ init__中的PendingDeprecationWarning(除非它杀死了) 为scikit-learn和朋友展示)，并放了一个大警告框 在文档顶部.这里的想法是不实际中断 任何人的代码，但开始传达出我们绝对的信息 不要以为任何人都可以使用它.

• 除了scipy.sparse之外，还有另一种方法:提高警告强度， 可能一直到FutureWarning，这样现有的脚本就不会 中断，但它们确实收到嘈杂的警告

• 最终，如果我们认为这样做会减少维护成本，请进行拆分 放入子包

(源).

现状

截至2018年5月(numpy 1.15，相关的拉动请求和提交)默认情况下，弃用警告(几乎总是)是无声的，因此大多数numpy的最终用户都不会看到此强烈提示.

最后，截至2018年11月的 numpy路线图提到多个相关主题，作为"任务和功能[numpy社区]之一，将在中投入资源":

NumPy内部的某些内容实际上与NumPy的范围不匹配.

• 用于numpy.fft的后端系统(因此，例如fft-mkl不需要猴子补丁numpy)
• 将掩码数组重写为不是ndarray子类-也许在单独的项目中?
• MaskedArray作为鸭子数组类型，和/或
• 支持缺失值的dtypes
• 编写一个策略，以处理linalg和fft的numpy和scipy之间的重叠(并实现).
• 弃用np.matrix

只要较大的库/许多用户(尤其是scipy.sparse)依赖于矩阵类，这种状态就可能保持不变.但是，有正在进行的讨论使scipy.sparse依赖于其他内容，例如 pydata/sparse .无论弃用过程的发展如何，用户都应在新代码中使用ndarray类，并且如果可能的话，最好移植旧代码.最终，矩阵类可能最终会放在单独的程序包中，以消除因其以当前形式存在而带来的一些负担.

What is the status of the matrix class in NumPy?

I keep being told that I should use the ndarray class instead. Is it worth/safe using the matrix class in new code I write? I don't understand why I should use ndarrays instead.

解决方案

tl; dr: the numpy.matrix class is getting deprecated. There are some high-profile libraries that depend on the class as a dependency (the largest one being scipy.sparse) which hinders proper short-term deprecation of the class, but users are strongly encouraged to use the ndarray class (usually created using the numpy.array convenience function) instead. With the introduction of the @ operator for matrix multiplication a lot of the relative advantages of matrices have been removed.

Why (not) the matrix class?

numpy.matrix is a subclass of numpy.ndarray. It was originally meant for convenient use in computations involving linear algebra, but there are both limitations and surprising differences in how they behave compared to instances of the more general array class. Examples for fundamental differences in behaviour:

• Shapes: arrays can have an arbitrary number of dimensions ranging from 0 to infinity (or 32). Matrices are always two-dimensional. Oddly enough, while a matrix can't be created with more dimensions, it's possible to inject singleton dimensions into a matrix to end up with technically a multidimensional matrix: np.matrix(np.random.rand(2,3))[None,...,None].shape == (1,2,3,1) (not that this is of any practical importance).
• Indexing: indexing arrays can give you arrays of any size depending on how you are indexing it. Indexing expressions on matrices will always give you a matrix. This means that both arr[:,0] and arr[0,:] for a 2d array gives you a 1d ndarray, while mat[:,0] has shape (N,1) and mat[0,:] has shape (1,M) in case of a matrix.
• Arithmetic operations: the main reason for using matrices in the old days was that arithmetic operations (in particular, multiplication and power) on matrices performs matrix operations (matrix multiplication and matrix power). The same for arrays results in elementwise multiplication and power. Consequently mat1 * mat2 is valid if mat1.shape[1] == mat2.shape[0], but arr1 * arr2 is valid if arr1.shape == arr2.shape (and of course the result means something completely different). Also, surprisingly, mat1 / mat2 performs elementwise division of two matrices. This behaviour is probably inherited from ndarray but makes no sense for matrices, especially in light of the meaning of *.
• Special attributes: matrices have a few handy attributes in addition to what arrays have: mat.A and mat.A1 are array views with the same value as np.array(mat) and np.array(mat).ravel(), respectively. mat.T and mat.H are the transpose and conjugate transpose (adjoint) of the matrix; arr.T is the only such attribute that exists for the ndarray class. Finally, mat.I is the inverse matrix of mat.

It's easy enough writing code that works either for ndarrays or for matrices. But when there's a chance that the two classes have to interact in code, things start to become difficult. In particular, a lot of code could work naturally for subclasses of ndarray, but matrix is an ill-behaved subclass that can easily break code that tries to rely on duck typing. Consider the following example using arrays and matrices of shape (3,4):

import numpy as np

shape = (3, 4)
arr = np.arange(np.prod(shape)).reshape(shape) # ndarray
mat = np.matrix(arr) # same data in a matrix
print((arr + mat).shape)           # (3, 4), makes sense
print((arr[0,:] + mat[0,:]).shape) # (1, 4), makes sense
print((arr[:,0] + mat[:,0]).shape) # (3, 3), surprising

Adding slices of the two objects is catastrophically different depending on the dimension along which we slice. Addition on both matrices and arrays happens elementwise when the shapes are the same. The first two cases in the above are intuitive: we add two arrays (matrices), then we add two rows from each. The last case is really surprising: we probably meant to add two columns and ended up with a matrix. The reason of course is that arr[:,0] has shape (3,) which is compatible with shape (1,3), but mat[:.0] has shape (3,1). The two are broadcast together to shape (3,3).

Finally, the largest advantage of the matrix class (i.e. the possibility to concisely formulate complicated matrix expressions involving a lot of matrix products) was removed when the @ matmul operator was introduced in python 3.5, first implemented in numpy 1.10. Compare the computation of a simple quadratic form:

v = np.random.rand(3); v_row = np.matrix(v)
arr = np.random.rand(3,3); mat = np.matrix(arr)

print(v.dot(arr.dot(v))) # pre-matmul style
# 0.713447037658556, yours will vary
print(v_row * mat * v_row.T) # pre-matmul matrix style
# [[0.71344704]]
print(v @ arr @ v) # matmul style
# 0.713447037658556

Looking at the above it's clear why the matrix class was widely preferred for working with linear algebra: the infix * operator made the expressions much less verbose and much easier to read. However, we get the same readability with the @ operator using modern python and numpy. Furthermore, note that the matrix case gives us a matrix of shape (1,1) which should technically be a scalar. This also implies that we can't multiply a column vector with this "scalar": (v_row * mat * v_row.T) * v_row.T in the above example raises an error because matrices with shape (1,1) and (3,1) can't be multiplied in this order.

For completeness' sake it should be noted that while the matmul operator fixes the most common scenario in which ndarrays are suboptimal compared to matrices, there are still a few shortcomings in handling linear algebra elegantly using ndarrays (although people still tend to believe that overall it's preferable to stick to the latter). One such example is matrix power: mat ** 3 is the proper third matrix power of a matrix (whereas it's the elementwise cube of an ndarray). Unfortunately numpy.linalg.matrix_power is quite more verbose. Furthermore, in-place matrix multiplication only works fine for the matrix class. In contrast, while both PEP 465 and the python grammar allow @= as an augmented assignment with matmul, this is not implemented for ndarrays as of numpy 1.15.

Deprecation history

Considering the above complications concerning the matrix class there have been recurring discussions of its possible deprecation for a long time. The introduction of the @ infix operator which was a huge prerequisite for this process happened in September 2015. Unfortunately the advantages of the matrix class in earlier days meant that its use spread wide. There are libraries that depend on the matrix class (one of the most important dependent is scipy.sparse which uses both numpy.matrix semantics and often returns matrices when densifying), so fully deprecating them has always been problematic.

Already in a numpy mailing list thread from 2009 I found remarks such as

numpy was designed for general purpose computational needs, not any one branch of math. nd-arrays are very useful for lots of things. In contrast, Matlab, for instance, was originally designed to be an easy front-end to linear algebra package. Personally, when I used Matlab, I found that very awkward -- I was usually writing 100s of lines of code that had nothing to do with linear algebra, for every few lines that actually did matrix math. So I much prefer numpy's way -- the linear algebra lines of code are longer an more awkward, but the rest is much better.

The Matrix class is the exception to this: is was written to provide a natural way to express linear algebra. However, things get a bit tricky when you mix matrices and arrays, and even when sticking with matrices there are confusions and limitations -- how do you express a row vs a column vector? what do you get when you iterate over a matrix? etc.

There has been a bunch of discussion about these issues, a lot of good ideas, a little bit of consensus about how to improve it, but no one with the skill to do it has enough motivation to do it.

These reflect the benefits and difficulties arising from the matrix class. The earliest suggestion for deprecation I could find is from 2008, although partly motivated by unintuitive behaviour that has changed since (in particular, slicing and iterating over a matrix will result in (row) matrices as one would most likely expect). The suggestion showed both that this is a highly controversial subject and that infix operators for matrix multiplication are crucial.

The next mention I could find is from 2014 which turned out to be a very fruitful thread. The ensuing discussion raises the question of handling numpy subclasses in general, which general theme is still very much on the table. There is also strong criticism:

What sparked this discussion (on Github) is that it is not possible to write duck-typed code that works correctly for:

• ndarrays
• matrices
• scipy.sparse sparse matrixes

The semantics of all three are different; scipy.sparse is somewhere between matrices and ndarrays with some things working randomly like matrices and others not.

With some hyberbole added, one could say that from the developer point of view, np.matrix is doing and has already done evil just by existing, by messing up the unstated rules of ndarray semantics in Python.

followed by a lot of valuable discussion of the possible futures for matrices. Even with no @ operator at the time there is a lot of thought given to the deprecation of the matrix class and how it might affect users downstream. As far as I can tell this discussion has directly led to the inception of PEP 465 introducing matmul.

In early 2015:

In my opinion, a "fixed" version of np.matrix should (1) not be a np.ndarray subclass and (2) exist in a third party library not numpy itself.

I don't think it's really feasible to fix np.matrix in its current state as an ndarray subclass, but even a fixed matrix class doesn't really belong in numpy itself, which has too long release cycles and compatibility guarantees for experimentation -- not to mention that the mere existence of the matrix class in numpy leads new users astray.

Once the @ operator had been available for a while the discussion of deprecation surfaced again, reraising the topic about the relationship of matrix deprecation and scipy.sparse.

Eventually, first action to deprecate numpy.matrix was taken in late November 2017. Regarding dependents of the class:

How would the community handle the scipy.sparse matrix subclasses? These are still in common use.

They're not going anywhere for quite a while (until the sparse ndarrays materialize at least). Hence np.matrix needs to be moved, not deleted.

(source) and

while I want to get rid of np.matrix as much as anyone, doing that anytime soon would be really disruptive.

• There are tons of little scripts out there written by people who didn't know better; we do want them to learn not to use np.matrix but breaking all their scripts is a painful way to do that

• There are major projects like scikit-learn that simply have no alternative to using np.matrix, because of scipy.sparse.

So I think the way forward is something like:

• Now or whenever someone gets together a PR: issue a PendingDeprecationWarning in np.matrix.__init__ (unless it kills performance for scikit-learn and friends), and put a big warning box at the top of the docs. The idea here is to not actually break anyone's code, but start to get out the message that we definitely don't think anyone should use this if they have any alternative.

• After there's an alternative to scipy.sparse: ramp up the warnings, possibly all the way to FutureWarning so that existing scripts don't break but they do get noisy warnings

• Eventually, if we think it will reduce maintenance costs: split it into a subpackage

(source).

Status quo

As of May 2018 (numpy 1.15, relevant pull request and commit) the matrix class docstring contains the following note:

It is no longer recommended to use this class, even for linear algebra. Instead use regular arrays. The class may be removed in the future.

And at the same time a PendingDeprecationWarning has been added to matrix.__new__. Unfortunately, deprecation warnings are (almost always) silenced by default, so most end-users of numpy will not see this strong hint.

Finally, the numpy roadmap as of November 2018 mentions multiple related topics as one of the "tasks and features [the numpy community] will be investing resources in":

Some things inside NumPy do not actually match the Scope of NumPy.

• A backend system for numpy.fft (so that e.g. fft-mkl doesn’t need to monkeypatch numpy)
• Rewrite masked arrays to not be a ndarray subclass – maybe in a separate project?
• MaskedArray as a duck-array type, and/or
• dtypes that support missing values
• Write a strategy on how to deal with overlap between numpy and scipy for linalg and fft (and implement it).
• Deprecate np.matrix

It's likely that this state will stay as long as larger libraries/many users (and in particular scipy.sparse) rely on the matrix class. However, there's ongoing discussion to move scipy.sparse to depend on something else, such as pydata/sparse. Irrespective of the developments of the deprecation process users should use the ndarray class in new code and preferably port older code if possible. Eventually the matrix class will probably end up in a separate package to remove some of the burdens caused by its existence in its current form.

这篇关于NumPy矩阵类的弃用状态的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！