pandas 数据帧的分位数归一化 [英] quantile normalization on pandas dataframe

问题描述

简而言之，如何在Python中的大型Pandas数据帧（大约2,000,000行）上应用分位数归一化？

Simply speaking, how to apply quantile normalization on a large Pandas dataframe (probably 2,000,000 rows) in Python?

PS。我知道有一个名为rpy2的程序包，可以在R中使用分位数归一化在子进程中运行R。但是事实是，当我使用如下数据集时，R无法计算正确的结果：

PS. I know that there is a package named rpy2 which could run R in subprocess, using quantile normalize in R. But the truth is that R cannot compute the correct result when I use the data set as below:

5.690386092696389541e-05,2.051450375415418849e-05,1.963190184049079707e-05,1.258362869906251862e-04,1.503352476021528139e-04,6.881341586355676286e-06
8.535579139044583634e-05,5.128625938538547123e-06,1.635991820040899643e-05,6.291814349531259308e-05,3.006704952043056075e-05,6.881341586355676286e-06
5.690386092696389541e-05,2.051450375415418849e-05,1.963190184049079707e-05,1.258362869906251862e-04,1.503352476021528139e-04,6.881341586355676286e-06
2.845193046348194770e-05,1.538587781561563968e-05,2.944785276073619561e-05,4.194542899687506431e-05,6.013409904086112150e-05,1.032201237953351358e-05

编辑：

我想要什么：

鉴于上面显示的数据，如何应用分位数归一化降低 https://en.wikipedia.org/wiki/Quantile_normalization 中的步骤。

Given the data shown above, how to apply quantile normalization following steps in https://en.wikipedia.org/wiki/Quantile_normalization.

我在Python中找到了一段代码，声明它可以计算分位数归一化：

I found a piece of code in Python declaring that it could compute the quantile normalization:

import rpy2.robjects as robjects
import numpy as np
from rpy2.robjects.packages import importr
preprocessCore = importr('preprocessCore')


matrix = [ [1,2,3,4,5], [1,3,5,7,9], [2,4,6,8,10] ]
v = robjects.FloatVector([ element for col in matrix for element in col ])
m = robjects.r['matrix'](v, ncol = len(matrix), byrow=False)
Rnormalized_matrix = preprocessCore.normalize_quantiles(m)
normalized_matrix = np.array( Rnormalized_matrix)

该代码可以很好地处理示例数据用在代码中，但是当我用上面给出的数据测试它时，结果出错了。

The code works fine with the sample data used in the code, however when I test it with the data given above the result went wrong.

因为ryp2为r提供了一个接口un R在python子进程中，我直接在R中再次对其进行了测试，结果仍然是错误的。结果，我认为原因是R中的方法不正确。

Since ryp2 provides an interface to run R in python subprocess, I test it again in R directly and the result was still wrong. As a result I think the reason is that the method in R is wrong.

推荐答案

好，我自己实现了相对较高的方法效率。

Ok I implemented the method myself of relatively high efficiency.

完成后，这种逻辑似乎很简单，但是无论如何，我决定将其发布在这里，因为任何人都感到困惑，就像我无法谷歌搜索时一样可用代码。

After finishing, this logic seems kind of easy but, anyway, I decided to post it here for any one feels confused like I was when I couldn't googled the available code.

代码在github中：分位数归一化

The code is in github: Quantile Normalize

这篇关于 pandas 数据帧的分位数归一化的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！