如何从多个URL中使用urllib.request.urlopen获取所有图像URL [英] How to get all image urls with urllib.request.urlopen from multiple urls

问题描述

from bs4 import BeautifulSoup
import urllib.request

urls = [
"https://archillect.com/1",
"https://archillect.com/2",
"https://archillect.com/3",
]

soup = BeautifulSoup(urllib.request.urlopen(urls))

for u in urls:
   for img in soup.find_all("img", src=True):
    print(img["src"])

AttributeError:列表"对象没有属性超时"

AttributeError: 'list' object has no attribute 'timeout'

推荐答案

@krishna给出了答案.我给您另一种解决方案仅供参考.

@krishna has given you the answer. I'll give you another solution for reference only.

from simplified_scrapy import Spider, SimplifiedDoc, SimplifiedMain, utils
class ImageSpider(Spider):
  name = 'archillect'
  start_urls = ["https://archillect.com/1","https://archillect.com/2","https://archillect.com/3"]
  def afterResponse(self, response, url, error=None, extra=None):
    try:
      # Create file name
      end = url.find('?') if url.find('?')>0 else len(url)
      name = 'data'+url[url.rindex('/',0,end):end]
      # save image
      if utils.saveResponseAsFile(response,name,'image'):
        return None 
      else:
        return Spider.afterResponse(self, response, url, error)
    except Exception as err:
      print (err)
  def extract(self,url,html,models,modelNames):
    doc = SimplifiedDoc(html)
    urls = doc.listImg(url=url.url)
    return {'Urls':urls} 
SimplifiedMain.startThread(ImageSpider()) # Start

还有更多示例: https://github.com/yiyedata/simplified-scrapy-demo/tree/master/spider_examples

这篇关于如何从多个URL中使用urllib.request.urlopen获取所有图像URL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！