从9gag刮取图像，无法读取正确的HTML页面 [英] Scrape images from 9gag, unable to read correct HTML-page
问题描述
我正在尝试编写一个脚本，该脚本将仅刮擦9gag的图像和图像.但是我遇到了一个问题，就是我的请求或Beautifulsoup收到了错误的HTML页面.Beautifulsoup当前正在获取源页面，而不是包含图像的页面.
为什么Beautifulsoup排除包含实际图像的类?还是不同的HTML页面?
Im trying to write a script thats gonna scrape 9gag for images and images only. But i have faced a problem which is that my requests or the Beautifulsoup is getting the wrong HTML page.
Beautifulsoup is currently getting the source-page and not the page that is containing the images.
Why is Beautifulsoup excluding the classes that contain the actual images? Or is it diffrent HTML-pages?
我曾尝试过为美丽的汤"解析器使用不同的格式，但页面仍然错误.
I have tried diffrent formats for the Beautiful soup "parser" but still getting the wrong page.
如果转到9gag并单击鼠标右键，然后单击检查"，则可以转到图像，然后单击该页面以使用脚本提取图像.
If you go to 9gag and right-click and the "inspect" you can get to the images, and the page to extract the images with script.
我的脚本:
import requests
from bs4 import BeautifulSoup
import os
def download_image(url, fileName): #save image function
path = os.path.join("imgs", fileName)
f = open(path, 'wb')
f.write(requests.get(url).content)
f.close()
def fetch_url(url): # fetching url
page = requests.get(url)
return page
def parse_html(htmlPage): #parsing the url
soup = BeautifulSoup(htmlPage, "html.parser")
return soup
def retrieve_jpg_urls(soup):
list_of_urls = soup.find_all('list') #classes wanted
parsed_urls = []
for index in range(len(list_of_urls)):
try:
parsed_urls.append(soup.find_all('img')[index].attrs['src']) #img wanted inside class
except:
next
return parsed_urls
def main():
htmlPage = fetch_url("https://9gag.com/")
soup = parse_html(htmlPage.content)
jpgUrls = retrieve_jpg_urls(soup)
for index in range(len(jpgUrls)):
try:
download_image(jpgUrls[index], "savedpic{}.jpg".format(index))
except:
print("failed to parse image with url {}".format(jpgUrls[index]))
print("")
if __name__ == "__main__":
main()
Beautifulsoup正在获得什么:
What Beautifulsoup is getting:
<!DOCTYPE html>
<html lang="en">
<head>
<title>9GAG: Go Fun The World</title>
<link href="https://assets-9gag-fun.9cache.com" rel="preconnect"/>
<link href="https://img-9gag-fun.9cache.com" rel="preconnect"/>
<link href="https://miscmedia-9gag-fun.9cache.com" rel="preconnect"/>
<link href="https://images-cdn.9gag.com/img/9gag-og.png" rel="image_src"/>
<link href="https://9gag.com/" rel="canonical"/>
<link href="android-app://com.ninegag.android.app/http/9gag.com/" rel="alternate"/>
<link href="https://assets-9gag-fun.9cache.com/s/fab0aa49/5aa8c9f45ee3dd77f0fdbe4812f1afcf5913a34e/static/dist/core/img/favicon.ico" rel="shortcut icon"/>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
<meta content="9GAG has the best funny pics, gifs, videos, gaming, anime, manga, movie, tv, cosplay, sport, food, memes, cute, fail, wtf photos on the internet!" name="description"/>
我想要以下内容:
<img src="https://img-9gag-fun.9cache.com/photo/aLgyG2V_460s.jpg" alt="There&#039;s genuine friend love there" style="min-height: 566.304px;">
推荐答案
尝试在页面上提取JSON:
Try extracting the JSON on the page:
import re
import json
# ...
res = requests.get(...)
html = res.content
m = re.search('JSON\.parse\((.*)\);</script>', html)
double_encoded = m.group(1)
encoded = json.loads(double_encoded)
parsed = json.loads(encoded)
images = [p['images']['image700']['url'] for p in parsed['data']['posts']]
print(images)
输出:
['https://img-9gag-fun.9cache.com/photo/abY9Wg8_460s.jpg', 'https://img-9gag-fun.9cache.com/photo/aLgy4o5_460s.jpg', 'https://img-9gag-fun.9cache.com/photo/aE2LVeM_460s.jpg', 'https://img-9gag-fun.9cache.com/photo/amBEGb4_700b.jpg', 'https://img-9gag-fun.9cache.com/photo/aKxrv56_460s.jpg', 'https://img-9gag-fun.9cache.com/photo/a5M8wXN_460s.jpg', 'https://img-9gag-fun.9cache.com/photo/aNY6QEv_700b.jpg', 'https://img-9gag-fun.9cache.com/photo/aYY2Deq_700b.jpg', 'https://img-9gag-fun.9cache.com/photo/aQR0AEw_460s.jpg', 'https://img-9gag-fun.9cache.com/photo/aLgy19P_700b.jpg']
这篇关于从9gag刮取图像，无法读取正确的HTML页面的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
