如何匹配运行时间太长的python中的所有键值对 [英] how can I match all the key value pair in python which running too long
本文介绍了如何匹配运行时间太长的python中的所有键值对的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
用户项目相似性和建议:
我正在创建一个表格,建议购买此商品的客户也购买了算法"
输入数据集
User-item affinity and recommendations :
I am creating a table which suggests "customers who bought this item also bought algorithm "
Input dataset
productId userId
Prod1 a
Prod1 b
Prod1 c
Prod1 d
prod2 b
prod2 c
prod2 a
prod2 b
prod3 c
prod3 a
prod3 d
prod3 c
prod4 a
prod4 b
prod4 d
prod4 a
prod5 d
prod5 a
需要输出
Product1 Product2 score
Prod1 prod3
Prod1 prod4
Prod1 prod5
prod2 Prod1
prod2 prod3
prod2 prod4
prod2 prod5
prod3 Prod1
prod3 prod2
Using code :
#Get list of unique items
itemList=list(set(main["productId"].tolist()))
#Get count of users
userCount=len(set(main["productId"].tolist()))
#Create an empty data frame to store item affinity scores for items.
itemAffinity= pd.DataFrame(columns=('item1', 'item2', 'score'))
rowCount=0
#For each item in the list, compare with other items.
for ind1 in range(len(itemList)):
#Get list of users who bought this item 1.
item1Users = main[main.productId==itemList[ind1]]["userId"].tolist()
#print("Item 1 ", item1Users)
#Get item 2 - items that are not item 1 or those that are not analyzed already.
for ind2 in range(ind1, len(itemList)):
if ( ind1 == ind2):
continue
#Get list of users who bought item 2
item2Users=main[main.productId==itemList[ind2]]["userId"].tolist()
#print("Item 2",item2Users)
#Find score. Find the common list of users and divide it by the total users.
commonUsers= len(set(item1Users).intersection(set(item2Users)))
score=commonUsers / userCount
#Add a score for item 1, item 2
itemAffinity.loc[rowCount] = [itemList[ind1],itemList[ind2],score]
rowCount +=1
#Add a score for item2, item 1. The same score would apply irrespective of the sequence.
itemAffinity.loc[rowCount] = [itemList[ind2],itemList[ind1],score]
rowCount +=1
#Check final result
itemAffinity
代码在示例数据集上运行得很好,但是
该代码花费的时间太长,无法在包含100,000行的数据集中运行.请帮助我优化代码.
the code is running perfectly fine on a sample dataset but
The code is taking too long to run in dataset containing 100,000 rows. Please help me optimize the code.
推荐答案
此处的关键是创建productId的笛卡尔积.参见下面的代码,
The key here is to create a cartesian product of productId. See code below,
result=(main.drop_duplicates(['productId','userId'])
.assign(cartesian_key=1)
.pipe(lambda x:x.merge(x,on='cartesian_key'))
.drop('cartesian_key',axis=1)
.loc[lambda x:(x.productId_x!=x.productId_y) & (x.userId_x==x.userId_y)]
.groupby(['productId_x','productId_y']).size()
.div(data['userId'].nunique()))
result
Prod1 prod2 0.75
Prod1 prod3 0.75
Prod1 prod4 0.75
Prod1 prod5 0.5
prod2 Prod1 0.75
prod2 prod3 0.5
prod2 prod4 0.5
prod2 prod5 0.25
prod3 Prod1 0.75
prod3 prod2 0.5
prod3 prod4 0.5
prod3 prod5 0.5
prod4 Prod1 0.75
prod4 prod2 0.5
prod4 prod3 0.5
prod4 prod5 0.5
prod5 Prod1 0.5
prod5 prod2 0.25
prod5 prod3 0.5
prod5 prod4 0.5
方法2
result = (df.groupby(['productId','userId']).size()
.clip(upper=1)
.unstack()
.assign(key=1)
.reset_index()
.pipe(lambda x:x.merge(x,on='key'))
.drop('key',axis=1)
.loc[lambda x:(x.productId_x!=x.productId_y)]
.set_index(['productId_x','productId_y'])
.pipe(lambda x:x.set_axis(x.columns.str.split('_',expand=True),axis=1,inplace=False))
.swaplevel(axis=1)
.pipe(lambda x:(x['x']+x['y']))
.fillna(0)
.div(2)
.mean(axis=1))
这篇关于如何匹配运行时间太长的python中的所有键值对的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文