如何使用 BeautifulSoup 从网页上阅读更多内容来抓取评论 [英] How to Scrape reviews with read more from Webpages using BeautifulSoup

问题描述

我试图从网站上抓取评论，但无法抓取带有阅读更多"选项的评论.我只能获取数据，直到阅读更多.我正在使用 BeautifulSoup.任何帮助表示赞赏.

I am trying to scrape reviews from a website and am not able to scrape reviews having a 'read more' option. I am only able to get data till read more. I am using BeautifulSoup. Any help is appreciated.

推荐答案

@user5444075 and @SIM

@user5444075 and @SIM

div.user-review p.lnhgt

上面的好像已经不存在了.

The above does not seem to exist anymore.

下面的xpath可以用来获取所有的href.

The below xpath can be used to get all the href.

//a[contains(@id,"lnkTitle")]/@href

使用上面的href向服务器发出新的请求，并获取rev-main-content类的文本

Make a new request to the server using the above href and get the text of the class rev-main-content

这篇关于如何使用 BeautifulSoup 从网页上阅读更多内容来抓取评论的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！