Splash lua 脚本进行多次点击和访问 [英] Splash lua script to do multiple clicks and visits

问题描述

我正在尝试抓取 Google 学术搜索结果 并获取与搜索匹配的每个结果的所有 BiBTeX 格式.现在我有一个带有 Splash 的 Scrapy 爬虫.我有一个 lua 脚本，它将单击引用"链接并在获取引用的 BibTeX 格式的 href 之前加载模式窗口.但是看到有多个搜索结果，因此有多个引用"链接，我需要单击它们并加载各个 BibTeX 页面.

I'm trying to crawl Google Scholar search results and get all the BiBTeX format of each result matching the search. Right now I have a Scrapy crawler with Splash. I have a lua script which will click the "Cite" link and load up the modal window before getting the href of the BibTeX format of the citation. But seeing that there are multiple search results and hence multiple "Cite" links, I need to click them all and load up the individual BibTeX pages.

这是我所拥有的:

import scrapy
from scrapy_splash import SplashRequest


class CiteSpider(scrapy.Spider):
    name = "cite"
    allowed_domains = ["scholar.google.com", "scholar.google.ae"]
    start_urls = [
        'https://scholar.google.ae/scholar?q="thermodynamics"&hl=en'
    ]

    script = """
        function main(splash)
          local url = splash.args.url
          assert(splash:go(url))
          assert(splash:wait(0.5))
          splash:runjs('document.querySelectorAll("a.gs_nph[aria-controls=gs_cit]")[0].click()')
          splash:wait(3)
          local href = splash:evaljs('document.querySelectorAll(".gs_citi")[0].href')
          assert(splash:go(href))
          return {
            html = splash:html(),
            png = splash:png(),
            href=href,
          }
        end
        """

    def parse(self, response):
        yield SplashRequest(self.start_urls[0], self.parse_bib,
                            endpoint="execute",
                            args={"lua_source": self.script})

    def parse_bib(self, response):
        filename = response.url.split("/")[-2] + '.html'
        with open(filename, 'wb') as f:
            f.write(response.css("body > pre::text").extract()[0])

我想我应该在执行 querySelectorAll 调用时将Cite"链接的索引传递到 lua 脚本中，但我似乎无法找到将另一个变量传递到功能.此外，我认为在获得 BibTeX 后，我将不得不做一些肮脏的 javascript history.back() 返回原始结果页面，但我觉得有一种更优雅的方法来处理这个.>

I'm thinking I should pass the index of the "Cite" link into the lua script when I perform the querySelectorAll call but I can't seem to find a way to pass another variable into the function. Also I assume I'll have to do some dirty javascript history.back() to return back to the original results page after getting the BibTeX but I feel there's a more elegant way to handle this.

推荐答案

好吧，我想出了一个有效的解决方案.首先，我们需要 Lua 脚本是可变的，所以我们将使它成为一个函数:

Okay so I hacked up a solution which works. First of all we'll need the Lua script to be mutable so we'll make it a function:

def script(n):
    _script = """
        function main(splash)
          local url = splash.args.url
          local href = ""
          assert(splash:go(url))
          assert(splash:wait(0.5))
          splash:runjs('document.querySelectorAll("a.gs_nph[aria-controls=gs_cit]")[{}].click()')
          splash:wait(3)
          href = splash:evaljs('document.querySelectorAll("a.gs_citi")[0].href')
          assert(splash:go(href))
          return {}
        end
        """.format(n, "{html=splash:html(),png=splash:png(), href=href,}")
    return _script

然后我不得不修改 parse 函数，以便它点击页面上的所有引用"链接.这样做的方法是遍历页面上所有匹配的引用"链接，然后单独单击每个链接.我让 Lua 脚本再次加载页面(这很脏，但我想不出任何其他方式)并单击查询的引用"链接的索引.它还必须发出重复的请求，因此为什么 dont_filter=True 存在:

I then had to modify the parse function so that it clicks all the "Cite" links on the page. The way to do that is to iterate through all the matching "Cite" links on the page and to click on each one individually. I made the Lua script load the page again (which is dirty but I can't think of any other way) and click on the index of the queried "Cite" link. Also it has to make duplicate requests hence why the dont_filter=True is there:

def parse(self, response):
        n = len(response.css("a.gs_nph[aria-controls=gs_cit]").extract())
        for i in range(n):
            yield SplashRequest(response.url, self.parse_bib,
                                endpoint="execute",
                                args={"lua_source": script(i)},
                                dont_filter=True)

希望这会有所帮助.

这篇关于Splash lua 脚本进行多次点击和访问的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！