使用python抓取AJAX电子商务网站 [英] Scraping AJAX e-commerce site using python
问题描述
我在使用BeautifulSoup 抓取电子商务网站时遇到问题.我做了一些谷歌搜索，但仍然无法解决问题.
I have a problem on scraping an e-commerce site using BeautifulSoup. I did some Googling but I still can't solve the problem.
请参考图片:
这是我尝试抓取的网站:" https ://shopee.com.my/search?keyword = h370m "
问题:
-
当我尝试在Google Chrome(F12)上打开Inspect Element时，可以看到产品名称，价格等信息.但是，当我运行python程序时，无法获得相同的代码，并且标记在python结果中.经过一番谷歌搜索，我发现该网站使用AJAX查询来获取数据.
任何人都可以通过抓取AJAX网站来帮助我找到获取这些产品数据的最佳方法吗?我想以表格形式显示数据.
Anyone can help me on the best methods to get these product's data by scraping an AJAX site? I would like to display the data in a table form.
我的代码:
import requests
from bs4 import BeautifulSoup
source = requests.get('https://shopee.com.my/search?keyword=h370m')
soup = BeautifulSoup(source.text, 'html.parser')
print(soup)
推荐答案
欢迎使用StackOverflow！您可以检查将ajax请求发送到的位置并进行复制.
Welcome to StackOverflow! You can inspect where the ajax request is being sent to and replicate that.
在这种情况下，请求 fake-useragent 之类的包，也可以仅硬编码该代理的字符串.
In this case the request goes to this api url. You can then use requests to perform a similar request. Notice however that this api endpoint requires a correct UserAgent header. You can use a package like fake-useragent or just hardcode a string for the agent.
import requests
# fake useragent
from fake_useragent import UserAgent
user_agent = UserAgent().chrome
# or hardcode
user_agent = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1468.0 Safari/537.36'
url = 'https://shopee.com.my/api/v2/search_items/?by=relevancy&keyword=h370m&limit=50&newest=0&order=desc&page_type=search'
resp = requests.get(url, headers={
'User-Agent': user_agent
})
data = resp.json()
products = data.get('items')
这篇关于使用python抓取AJAX电子商务网站的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
