火花源主元分析中最佳特征个数的确定 [英] How to determine the optimum number of features in pca with pyspark
本文介绍了火花源主元分析中最佳特征个数的确定的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
通过SCI-KIT学习,我们可以根据累积方差图确定希望保留的功能数量,如下所示
from sklearn.decomposition import PCA
pca = PCA() # init pca
pca.fit(dataset) # fit the dataset into pca model
pca.explained_variance_ratio # this attribute shows how much variance is explained by each of the seven individual component
we can plot the cumulative value as below
plt.figure(figsize= (10, 8)) # size of the chart(size of the vectors)
cumulativeValue = pca.explained_variance_ratio_.cumsum() # get the cumulative sum
plt.plot(range(1,8), cumulativeValue, marker = 'o', linestyle="--")
我的问题是如何确定使用pysppark的最佳功能数量
推荐答案
我们可以通过以下explainedVariance
我是如何做到的来确定这一点。
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.feature import PCA
# used vector assembler to create the input the vector
vectorAssembler = VectorAssembler(inputCols=['inputCol1', 'inputCol2', 'inputCol3', 'inputCol4'], outputCol='pcaInput')
df = vectorAssembler.transform(dataset) # fetch data into vector assembler
pca = PCA(k=8, inputCol="pcaInput", outputCol="features") # here I Have defined maximum number of features that I have
pcaModel = pca.fit(df) # fit the data to pca to make the model
print(pcaModel.explainedVariance) # here it will explain the variances
cumValues = pcaModel.explainedVariance.cumsum() # get the cumulative values
# plot the graph
plt.figure(figsize=(10,8))
plt.plot(range(1,9), cumValues, marker = 'o', linestyle='--')
plt.title('variance by components')
plt.xlabel('num of components')
plt.ylabel('cumulative explained variance')
选择接近80%的参数数量
因此,在这种情况下,参数的最佳数量为2
这篇关于火花源主元分析中最佳特征个数的确定的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文