试图从网站上抓取电子邮件地址 [英] Trying to scrape email address from website

问题描述

我试图抓取这个网站:

www.united-church.ca/search/locator/all?keyw=&mission_units_ucc_ministry_type_advanced=10&locll=

我确实使用 Scrapy 抓取了它，但我无法抓取电子邮件地址.有人可以帮我吗?

I did scrape it using Scrapy, but I couldn't scrape email addresses. Can anyone help me?

这是我目前的代码:

# -*- coding: utf-8 -*-
import scrapy
from ..items import ChurchItem


class ChurchSpiderSpider(scrapy.Spider):
    name = 'church_spider'
    page_number = 1
    start_urls = ['https://www.united-church.ca/search/locator/all?keyw=&mission_units_ucc_ministry_type_advanced=10&locll=']

    def parse(self, response):
        items = ChurchItem()
        container = response.css(".icon-ministry")
        for t in container:
            church_name = t.css(".field-name-locator-ministry-title a::text").extract()
            church_phone = t.css(".field-name-field-phone::text").extract()
            church_address = t.css(".thoroughfare::text").extract()
            church_email = t.css(".field-name-field-mu-email span::text").extract()

            items["church_name"] = church_name
            items["church_phone"] = church_phone
            items["church_address"] = church_address
            items["church_email"] = church_email

            yield items

        # next_page = 'https://www.united-church.ca/search/locator/all?keyw=&mission_units_ucc_ministry_type_advanced=10&locll=&page=' + str(ChurchSpiderSpider.page_number)
        # if ChurchSpiderSpider.page_number <= 110:
        #     ChurchSpiderSpider.page_number += 1
        #     yield response.follow(next_page, callback=self.parse)

我找到了部分解决方案，但仍然不完整.输出现在是这样的:

I have found a partial solution, but it is still not complete. The output is now like:

{'church_address': ['7763 Highway 21'],
 'church_email': ['herbklaehn', ' [at] ', 'gmail.com'],
 'church_name': ['Allenford United Church'],
 'church_phone': ['519-35-6232']}

如何将 [at] 替换为 @ 并将电子邮件地址合并为一个字符串?

How do I replace [at] with @ and combine the email address into one string?

推荐答案

您可以尝试使用 Selenium 进行网页抓取，我尝试了此代码并给出了完美的结果.

You can try webscraping using Selenium, I tried this code and its giving perfect results.

from selenium import webdriver
from bs4 import BeautifulSoup


driver = webdriver.Chrome("chromedriver")
driver.get("https://www.united-church.ca/search/locator/all?keyw=&mission_units_ucc_ministry_type_advanced=10&locll=")

content = driver.page_source
soup = BeautifulSoup(content)

for all_emails in soup.find_all('a',class_="spamspan"):
    print(all_emails.text)

结果:

[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]

这篇关于试图从网站上抓取电子邮件地址的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！