Web Scraping - 从 Lazada 电子商务平台提取 URL [英] Web Scraping - URL extraction from Lazada ecommerce platform

问题描述

我目前正在尝试从 Lazada 电子商务平台抓取产品网址，但是我从网站而不是产品链接中获得了随机链接.

这是我需要的链接部分，我想列出产品页面上的所有产品 URL.

我希望你们能帮我解决这个问题，我知道这很简单它似乎不起作用，从昨天开始一直在看这个.

解决方案

页面是动态的.在 html 源代码中是生成产品 json 格式的脚本.您可以提取它，然后解析 json 对象以打印出网址:

from bs4 import BeautifulSoup, SoupStrainer进口请求导入jsonurl = "https://www.lazada.com.my/oldtown-white-coffee/?langFlag=en&q=All-Products&from=wangpu&pageTypeId=2"页面 = requests.get(url)数据 = 页.文本汤 = BeautifulSoup(数据)脚本 = 汤.find_all('脚本')jsonObj = 无对于脚本中的脚本:如果 'window.pageData=' 在 script.text 中:jsonStr = script.textjsonStr = jsonStr.split("window.pageData=")[1]jsonObj = json.loads(jsonStr)产品 = jsonObj['mods']['listItems']对于产品中的项目:打印(项目['productUrl'])

输出:

//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-classic-15s-x6-packs-free-oldtown-4-棒-随机-风味-i434054933-s631848411.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-x6-packs-i436317818-s639072921.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-x4-packs-i436315786-s639051449.html?mp=1//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-extra-rich-15s-x6-packs-free-oldtown-brown-mug-x1-4-sticks-random-flavor-i434056795-s631848533.html?mp=1//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-2-in-1-coffee-creamer-15s-x6-packs-free-oldtown-brown-mug-x1-4-sticks-random-flavor-i434061102-s631849047.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-x2-packs-free-2-sticks-random-flavor-i434043717-s631808896.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-x4-packs-i436302855-s639061262.html?mp=1//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-less-sugar-15s-x6-packs-free-oldtown-brown-mug-x1-4-sticks-random-flavor-i434054922-s631844581.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-extra-rich-15s-x4-packs-i436302850-s639053701.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-i428864462-s623547178.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-extra-rich-15s-x2-packs-free-2-sticks-random-flavor-i434051011-s631820678.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-i429053474-s623935676.html?mp=1//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-i429057450-s623942383.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x6-packs-i436315838-s639079875.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-mocha-15s-i429056707-s623944523.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-i446475373-s668555737.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-x2-packs-free-2-sticks-random-flavor-i434038995-s631814134.html?mp=1//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-hazelnut-15s-x6-packs-free-oldtown-4-sticks-random-flavor-i434059429-s631843615.html?mp=1//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-x2-packs-free-2-sticks-random-flavor-i434048248-s631818679.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-2-in-1-coffee-creamer-15s-x2-packs-free-2-sticks-random-flavor-i434045440-s631815896.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x4-packs-i436347739-s639057542.html?mp=1//www.lazada.com.my/products/oldtown-black-series-enrich-freeze-dried-instant-coffee-100g-i434617928-s633601781.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-x4-packs-i436353732-s639059385.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-2-in-1-coffee-creamer-15s-i429056294-s623942054.html?mp=1//www.lazada.com.my/products/bundle-of-4-oldtown-white-coffee-2-in-1-coffee-creamer-15s-x4-packs-free-4-sticks-random-flavor-i434053005-s631824941.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-x2-packs-free-2-sticks-random-flavor-i434041889-s631816196.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-extra-rich-15s-i429055995-s623946380.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-x6-packs-i436317821-s639075740.html?mp=1//www.lazada.com.my/products/oldtown-black-series-enrich-freeze-dried-instant-coffee-100g-2-glass-i435467037-s636334815.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x2-packs-free-2-sticks-random-flavor-i434043759-s631818279.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-2-in-1-coffee-creamer-15s-x2-packs-i436315754-s639016791.html?mp=1//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-x6-packs-i436353749-s639073757.html?mp=1//www.lazada.com.my/products/bundle-of-6-oldtown-3-in-1-white-milk-tea-13s-x6-packs-free-oldtown-4-sticks-random-flavor-i434058314-s631843262.html?mp=1//www.lazada.com.my/products/bundle-of-4-oldtown-white-coffee-3-in-1-hazelnut-15s-x4-packs-free-4-sticks-random-flavor-i446532232-s668570216.html?mp=1//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-x6-packs-i436303866-s639071227.html?mp=1//www.lazada.com.my/products/oldtown-black-series-enrich-freeze-dried-instant-coffee-100g-2-glass-foc-glass-mug-i442451196-s654637474.html?mp=1//www.lazada.com.my/products/bundle-of-4-oldtown-3-in-1-white-milk-tea-13s-x4-packs-free-4-sticks-random-flavor-i434046913-s631825874.html?mp=1//www.lazada.com.my/products/bundle-of-4-oldtown-white-coffee-3-in-1-classic-15s-x4-packs-free-4-sticks-random-flavor-i434052193-s631832149.html?mp=1//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-x2-packs-i436282923-s639018195.html?mp=1//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x6-packs-free-oldtown-4-sticks-random-flavor-i434062035-s631849174.html?mp=1

I am currently trying to scrape the products URLs from Lazada ecommerce platform, however i am getting random links from the website rather than the products links.

https://www.lazada.com.my/oldtown-white-coffee/?langFlag=en&q=All-Products&from=wangpu&pageTypeId=2

My code below:

from bs4 import BeautifulSoup, SoupStrainer
import requests

url = "https://www.lazada.com.my/oldtown-white-coffee/?langFlag=en&q=All- 
       Products&from=wangpu&pageTypeId=2"

page = requests.get(url)    
data = page.text
soup = BeautifulSoup(data)


links = soup.find_all('div', {'class': 'c16H9d'})

for link in soup.find_all("a"):
    print(link.get("href"))

The result I am getting out of this code(which is not what i want) :

This is the section of the links that I need, i wanted to list down all the products URLs from the products page.

I hope you guys can help me on this, I know it is simple it just doesn't seems to work, have been looking at this since yesterday.

解决方案

The page is dynamic. Within the html source code is the script that generates a json format of the products. You can pull that, then parse the json object to print off the urls:

from bs4 import BeautifulSoup, SoupStrainer
import requests
import json

url = "https://www.lazada.com.my/oldtown-white-coffee/?langFlag=en&q=All-Products&from=wangpu&pageTypeId=2"

page = requests.get(url)    
data = page.text
soup = BeautifulSoup(data)


scripts = soup.find_all('script')

jsonObj = None
for script in scripts:
    if 'window.pageData=' in script.text:
        jsonStr = script.text

        jsonStr = jsonStr.split("window.pageData=")[1]
        jsonObj = json.loads(jsonStr)


products = jsonObj['mods']['listItems']

for item in products:
    print (item['productUrl'])

Output:

//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-classic-15s-x6-packs-free-oldtown-4-sticks-random-flavor-i434054933-s631848411.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-x6-packs-i436317818-s639072921.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-x4-packs-i436315786-s639051449.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-extra-rich-15s-x6-packs-free-oldtown-brown-mug-x1-4-sticks-random-flavor-i434056795-s631848533.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-2-in-1-coffee-creamer-15s-x6-packs-free-oldtown-brown-mug-x1-4-sticks-random-flavor-i434061102-s631849047.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-x2-packs-free-2-sticks-random-flavor-i434043717-s631808896.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-x4-packs-i436302855-s639061262.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-less-sugar-15s-x6-packs-free-oldtown-brown-mug-x1-4-sticks-random-flavor-i434054922-s631844581.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-extra-rich-15s-x4-packs-i436302850-s639053701.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-i428864462-s623547178.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-extra-rich-15s-x2-packs-free-2-sticks-random-flavor-i434051011-s631820678.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-i429053474-s623935676.html?mp=1
//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-i429057450-s623942383.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x6-packs-i436315838-s639079875.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-mocha-15s-i429056707-s623944523.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-hazelnut-15s-i446475373-s668555737.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-x2-packs-free-2-sticks-random-flavor-i434038995-s631814134.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-hazelnut-15s-x6-packs-free-oldtown-4-sticks-random-flavor-i434059429-s631843615.html?mp=1
//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-x2-packs-free-2-sticks-random-flavor-i434048248-s631818679.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-2-in-1-coffee-creamer-15s-x2-packs-free-2-sticks-random-flavor-i434045440-s631815896.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x4-packs-i436347739-s639057542.html?mp=1
//www.lazada.com.my/products/oldtown-black-series-enrich-freeze-dried-instant-coffee-100g-i434617928-s633601781.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-x4-packs-i436353732-s639059385.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-2-in-1-coffee-creamer-15s-i429056294-s623942054.html?mp=1
//www.lazada.com.my/products/bundle-of-4-oldtown-white-coffee-2-in-1-coffee-creamer-15s-x4-packs-free-4-sticks-random-flavor-i434053005-s631824941.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-x2-packs-free-2-sticks-random-flavor-i434041889-s631816196.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-extra-rich-15s-i429055995-s623946380.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-less-sugar-15s-x6-packs-i436317821-s639075740.html?mp=1
//www.lazada.com.my/products/oldtown-black-series-enrich-freeze-dried-instant-coffee-100g-2-glass-i435467037-s636334815.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x2-packs-free-2-sticks-random-flavor-i434043759-s631818279.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-2-in-1-coffee-creamer-15s-x2-packs-i436315754-s639016791.html?mp=1
//www.lazada.com.my/products/oldtown-white-coffee-3-in-1-classic-15s-x6-packs-i436353749-s639073757.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-3-in-1-white-milk-tea-13s-x6-packs-free-oldtown-4-sticks-random-flavor-i434058314-s631843262.html?mp=1
//www.lazada.com.my/products/bundle-of-4-oldtown-white-coffee-3-in-1-hazelnut-15s-x4-packs-free-4-sticks-random-flavor-i446532232-s668570216.html?mp=1
//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-x6-packs-i436303866-s639071227.html?mp=1
//www.lazada.com.my/products/oldtown-black-series-enrich-freeze-dried-instant-coffee-100g-2-glass-foc-glass-mug-i442451196-s654637474.html?mp=1
//www.lazada.com.my/products/bundle-of-4-oldtown-3-in-1-white-milk-tea-13s-x4-packs-free-4-sticks-random-flavor-i434046913-s631825874.html?mp=1
//www.lazada.com.my/products/bundle-of-4-oldtown-white-coffee-3-in-1-classic-15s-x4-packs-free-4-sticks-random-flavor-i434052193-s631832149.html?mp=1
//www.lazada.com.my/products/oldtown-3-in-1-white-milk-tea-13s-x2-packs-i436282923-s639018195.html?mp=1
//www.lazada.com.my/products/bundle-of-6-oldtown-white-coffee-3-in-1-natural-cane-sugar-15s-x6-packs-free-oldtown-4-sticks-random-flavor-i434062035-s631849174.html?mp=1

这篇关于Web Scraping - 从 Lazada 电子商务平台提取 URL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！