相似性分数用于过滤 pandas 中的数据帧列 [英] Similarity score to filter dataframe columns in pandas

问题描述

我有一个 pandas 数据帧df，其列名如下

columns = ['Baillie Gifford Positive Change Fund B Accumulation',
 'Stewart Investors Worldwide Select Fund Class B (accumulation) Gbp',
 'Stewart Investors Worldwide Select Fund Class A (accumulation) Gbp',
 'Close Ftse Techmark Fund X Acc',
 'Stewart Investors Asia Pacific Leaders Fund Class B (accumulation) Gbp',
 'Stewart Investors Asia Pacific Leaders Fund Class A (accumulation) Gbp',
 'Stewart Investors Worldwide Sustainability Fund Class A (accumulation) Gbp',
 'Stewart Investors Worldwide Sustainability Fund Class B (accumulation) Gbp',
 'Mi Somerset Emerging Markets Dividend Growth A Accumulation Shares',
 'Axa Framlington Biotech Fund Gbp Z Acc',
 'Stewart Investors Global Emerging Markets Sustainability Fund Class B (accumulation) Gbp',
 'Schroder Asian Income Fund L Accumulation Gbp',
 'Fidelity Active Strategy - Fast - Asia Fund Y-acc-gbp',
 'Lf Miton Uk Value Opportunities Fund B Institutional Accumulation',
 'Liontrust India Fund C Acc Gbp',
 'Fidelity Asian Dividend Fund W Acc',
 'Stewart Investors Global Emerging Markets Sustainability Fund Class A (accumulation) Gbp',
 'Quilter Investors Emerging Markets Equity Growth Fund U2 (gbp) Accumulation',
 'Man Glg Continental European Growth Fund Retail Accumulation Shares (class A)',
 'Quilter Investors Europe (ex Uk) Equity Growth Fund A (gbp) Accumulation']

我想要的是筛选相似的列并保留其中一列。

例如'Stewart Investors Worldwide Select Fund Class B (accumulation) Gbp'，类似于'Stewart Investors Worldwide Select Fund Class A (accumulation) Gbp'，

我在想，NLP中用来识别相似文本的一些相似性分数在这里可能会有所帮助。但我不知道如何申请我的情况。

预期结果应该是保存一个相似文本的列表(我将用它来过滤我的数据帧)。例如：

columns_filtered = ['Baillie Gifford Positive Change Fund B Accumulation',
 'Stewart Investors Worldwide Select Fund Class B (accumulation) Gbp',
 'Close Ftse Techmark Fund X Acc',
 'Stewart Investors Asia Pacific Leaders Fund Class A (accumulation) Gbp',
 'Stewart Investors Worldwide Sustainability Fund Class B (accumulation) Gbp',
 'Mi Somerset Emerging Markets Dividend Growth A Accumulation Shares',
 'Axa Framlington Biotech Fund Gbp Z Acc',
 'Stewart Investors Global Emerging Markets Sustainability Fund Class B (accumulation) Gbp',
 'Schroder Asian Income Fund L Accumulation Gbp',
 'Fidelity Active Strategy - Fast - Asia Fund Y-acc-gbp',
 'Lf Miton Uk Value Opportunities Fund B Institutional Accumulation',
 'Liontrust India Fund C Acc Gbp',
 'Fidelity Asian Dividend Fund W Acc',
 'Stewart Investors Global Emerging Markets Sustainability Fund Class A (accumulation) Gbp',
 'Quilter Investors Emerging Markets Equity Growth Fund U2 (gbp) Accumulation',
 'Man Glg Continental European Growth Fund Retail Accumulation Shares (class A)',
 'Quilter Investors Europe (ex Uk) Equity Growth Fund A (gbp) Accumulation']

有帮助吗？

推荐答案

我找到了解决方案

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity 
import numpy as np


vectorizer = CountVectorizer().fit_transform(df.columns.tolist())
vector = vectorizer.toarray()

similarity_score = cosine_similarity(vector)


df_similarity = pd.DataFrame(np.asmatrix(similarity_score))
df_similarity.columns = df.columns
df_similarity.index = df.columns
df_similarity

df_similarity是一个数据框，其中保存每个列名与其他列名的相似性索引。

请注意，我使用了NLP中使用的一个相似性分数。用户可以使用任何可能的相似性分数。

这篇关于相似性分数用于过滤 pandas 中的数据帧列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！