使用BeautifulSoup在python中进行Web抓取-如何转置结果? [英] Web scraping in python using BeautifulSoup - how to transpose results?

问题描述

我在下面构建了代码，并遇到了如何转置结果的问题.实际上，我正在寻找以下结果:

I built the code below and am having issues of how to transpose the results. Effectively I am looking for the following result:

#    Column headers: 'company name',  'Work/Life Balance',   'Salary/Benefits',  'Job Security/Advancement', 'Management', 'Culture'  
#    Row 1: 3M, 3.8, 3.9, 3.5, 3.6, 3.8
#    Row 2: Google, . . .

当前发生的情况如下:

#    Column headers: 'Name', 'Rating', 'Category'
#    Row 1: 3M, 3.8, Work/Life Balance
#    Row 2: 3M, 3.9, Salary/Benefits
#    and so on . . .

到目前为止，我的代码:

My code thus far:

import  requests
import pandas as pd
from bs4 import BeautifulSoup


number = []
category = []
name = []
company = ['3M', 'Google']
for company_name in company:
    try:
        url = 'https://ca.indeed.com/cmp/'+company_name
        page = requests.get(url)
        soup = BeautifulSoup(page.content, 'html.parser')
        rating = soup.find(class_='cmp-ReviewAndRatingsStory-rating')
        rating = rating.find('tbody')
        rows = rating.find_all('tr')
    except:
        pass
    for row in rows:
        number.append(str(row.find_all('td')[0].text))
        category.append(str(row.find_all('td')[2].text))
        name.append(company_name)
    cols = {'Name':name,'Rating':number,'Category':category}
    df = pd.DataFrame(cols)
    print(df)

代码产生的内容:

      Name Rating                  Category
0       3M    3.8         Work/Life Balance
1       3M    3.9           Salary/Benefits
2       3M    3.5  Job Security/Advancement
3       3M    3.6                Management
4       3M    3.8                   Culture
5   Google    4.2         Work/Life Balance
6   Google    4.0           Salary/Benefits
7   Google    3.6  Job Security/Advancement
8   Google    3.9                Management
9   Google    4.2                   Culture
10   Apple    3.8         Work/Life Balance
11   Apple    4.1           Salary/Benefits
12   Apple    3.7  Job Security/Advancement
13   Apple    3.7                Management
14   Apple    4.1                   Culture

使用以下代码复制结果:

replicate result by using code below:

import pandas as pd
name = ['3M','3M','3M','3M','3M','Google','Google','Google','Google','Google','Apple','Apple','Apple','Apple','Apple']
number = ['3.8','3.9','3.5','3.6','3.8','4.2','4.0','3.6','3.9','4.2','3.8','4.1','3.7','3.7','4.1']
category = ['Work/Life Balance',' Salary/Benefits','Job Security/Advancement','Management','Culture','Work/Life Balance',' Salary/Benefits','Job Security/Advancement','Management','Culture','Work/Life Balance',' Salary/Benefits','Job Security/Advancement','Management','Culture']
cols = {'Name':name,'Rating':number,'Category':category}
df = pd.DataFrame(cols)
print(df)

推荐答案

这是一种可能的方法.

import pandas as pd
name = ['3M','3M','3M','3M','3M','Google','Google','Google','Google','Google','Apple','Apple','Apple','Apple','Apple']
number = ['3.8','3.9','3.5','3.6','3.8','4.2','4.0','3.6','3.9','4.2','3.8','4.1','3.7','3.7','4.1']
category = ['Work/Life Balance',' Salary/Benefits','Job Security/Advancement','Management','Culture','Work/Life Balance',' Salary/Benefits','Job Security/Advancement','Management','Culture','Work/Life Balance',' Salary/Benefits','Job Security/Advancement','Management','Culture']
cols = {'Name':name,'Rating':number,'Category':category}
df = pd.DataFrame(cols)
print(df)



from collections import defaultdict
aggregated_data = defaultdict(dict)
for idx, row in df.iterrows():
    aggregated_data[row.Name][row.Category] = row.Rating

result = pd.DataFrame(aggregated_data).T
print(result)

结果:

        Salary/Benefits Culture Job Security/Advancement Management Work/Life Balance
3M                  3.9     3.8                      3.5        3.6               3.8
Google              4.0     4.2                      3.6        3.9               4.2
Apple               4.1     4.1                      3.7        3.7               3.8

我认为这不是惯用的"方法.由于它使用本机Python数据类型和循环，因此它可能比纯熊猫解决方案要慢得多.但是，如果您的数据不是那么大，也许可以.

I don't think this is the "idiomatic" approach. Since it uses native Python data types and loops, it's probably considerably slower than a pure pandas solution. But if your data isn't that big, maybe that's OK.

我认为在最后一步中转置会导致列名以令人惊讶的顺序放置，因此这是一种从字典列表构造最终数据帧的方法.

I think transposing in that last step there is causing the column names to get put in a surprising order, so here's an approach that constructs the final dataframe from a list of dicts instead.

from collections import defaultdict
data_by_name = defaultdict(dict)
for idx, row in df.iterrows():
    data_by_name[row.Name][row.Category] = row.Rating

aggregated_rows = [{"company name": name, **ratings} for name, ratings in data_by_name.items()]
result = pd.DataFrame(aggregated_rows)
print(result)

结果:

  company name Work/Life Balance  Salary/Benefits Job Security/Advancement Management Culture
0           3M               3.8              3.9                      3.5        3.6     3.8
1       Google               4.2              4.0                      3.6        3.9     4.2
2        Apple               3.8              4.1                      3.7        3.7     4.1

这篇关于使用BeautifulSoup在python中进行Web抓取-如何转置结果?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！