HtmlUnit和XPath:DOMNode.getByXPath仅适用于HtmlPage吗? [英] HtmlUnit and XPath: DOMNode.getByXPath only works on HtmlPage?

问题描述

我正在尝试解析页面，其中包含指向其文章的链接重要内容如下:

I'm trying to parse a page with links to articles whose important content looks like this:

<div class="article">
  <h1 style="float: none;"><a href="performing-arts">Performing Arts</a></h1>
  <a href="/performing-arts/EIF-theatre-review-Sin-Sangre.6517348.jp">
    <span class="mth3">
      <span id="wctlMiniTemplate1_ctl00_ctl00_ctl01_WctlPremiumContentIcon1">                               
      </span>
      EIF theatre review: Sin Sangre | The Man Who Fed Butterflies | Caledonia | Songs Of Ascension | Vieux Carré | The Gospel At Colonus
    </span>
    <span class="mtp">The EIF&#39;s theatre programme wasn&#39;t as far-reaching as it could have been, but did find an exoticism in the familiar,  writes Mark Fisher </span>
  </a>                  
</div>

这是在Java中使用HtmlUnit和XPath进行最小程度的刮擦的情况(为简洁起见，删除了导入):

Here is a minimal scraping case in Java using HtmlUnit and XPath (imports removed for brevity):

public class MinimalTest {
    public static void main(String[] args) throws Exception {
        WebClient client = new WebClient();
        client.setJavaScriptEnabled(false);
        client.setCssEnabled(false);
        System.out.println("Fetching front page");
        HtmlPage frontPage = client.getPage("http://living.scotsman.com/sectionhome.aspx?sectionID=7063");
        List<ArticleInfo> articleInfos = extractArticleInfo(frontPage);

        for (ArticleInfo info : articleInfos)
        {
            System.out.println("Title: " + info.getTitle());
            System.out.println("Intro: " + info.getFirstPara());
            System.out.println("Link: " + info.getLink());
        }
    }

    @SuppressWarnings("unchecked") // xpath returns List<?>
    private static List<ArticleInfo> extractArticleInfo(HtmlPage frontPage) {
        System.out.println("Extracting article links");
        List<HtmlDivision> articleDivs = (List<HtmlDivision>) frontPage.getByXPath("//div[@class='article']");
        System.out.println(String.format("Found %d articles", articleDivs.size()));
        List<ArticleInfo> articleLinks = new ArrayList<ArticleInfo>(articleDivs.size());
        for (HtmlDivision div : articleDivs) {
            articleLinks.add(ArticleInfo.constructFromArticleDiv(div));
        }
        return articleLinks;
    }

    private static class ArticleInfo {
        private final String title;
        private final String link;
        private final String firstPara;

        public ArticleInfo(final String link, final String title, final String firstPara) {
            this.link = link;
            this.title = title;
            this.firstPara = firstPara;
        }
        public static ArticleInfo constructFromArticleDiv(final HtmlDivision div) {
            String link = ((DomText) div.getFirstByXPath("//a/@href/text()")).asText();
            String title = ((DomText) div.getFirstByXPath("//span[@class='mth3']/text()")).asText();
            String firstPara = ((DomText) div.getFirstByXPath("//span[@class='mtp']/text()")).asText();
            return new ArticleInfo(link, title, firstPara);
        }
        public String getTitle() {
            return title;
        }
        public String getFirstPara() {
            return firstPara;
        }
        public String getLink() {
            return link;
        }
    }  
}

我期望的输出:

Title: EIF theatre review: Sin Sangre | The Man Who Fed Butterflies | Caledonia | Songs Of Ascension | Vieux Carré | The Gospel At Colonus 
Intro: The EIF's theatre programme wasn't as far-reaching as it could have been, but did find an exoticism in the familiar, writes Mark Fisher 
Link: http://living.scotsman.com/performing-arts/EIF-theatre-review-Sin-Sangre.6517348.jp

我得到的东西:

Fetching front page
Extracting article links
Found 24 articles
Exception in thread "main" java.lang.NullPointerException
    at com.allthefestivals.app.crawler.MinimalTest$ArticleInfo.constructFromArticleDiv(MinimalTest.java:68)
    at com.allthefestivals.app.crawler.MinimalTest.extractArticleInfo(MinimalTest.java:50)
    at com.allthefestivals.app.crawler.MinimalTest.main(MinimalTest.java:30)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)

在HtmlPage上调用getByXPath可以正常工作，但在其他任何HtmlElement上似乎均不返回任何内容.怎么了?这是HtmlUnit中的错误还是实现上的空白，还是我缺少有关XPath语法的细微之处?

Calling getByXPath works fine on a HtmlPage but seems to return nothing on any other HtmlElement. What's wrong? Is this a bug or implementation gap in HtmlUnit, or am I missing something subtle about XPath syntax?

其解决方案不适用于我的相关问题: XPath _relative_到HTMLUnit/Groovy中的给定元素?

Related question whose solution didn't work for me: XPath _relative_ to given element in HTMLUnit/Groovy?

推荐答案

您已尝试将属性视为元素.尝试以下方法:

You've tried to treat an attribute as an element. Try this instead:

String link = ((DomAttr) div.getFirstByXPath("//a/@href")).getValue();

然后我得到了

Fetching front page
Extracting article links
Found 24 articles
Title: EIF theatre review: Sin Sangre | The Man Who Fed Butterflies | Caledonia | Songs Of Ascension | Vieux Carré | The Gospel At Colonus
Intro: The EIF's theatre programme wasn't as far-reaching as it could have been, but did find an exoticism in the familiar, writes Mark Fisher
Link: /Register.aspx?ReturnURL=http%3a%2f%2fliving.scotsman.com%2fsectionhome.aspx%3fsectionID%3d7063
...

此外，您的ArticleInfo类将"link"声明为String，然后为其分配一些(自定义?)类.为了使它能够编译，我不得不稍作改动.

Also, your ArticleInfo class declares "link" to be a String, then assigns it some (custom?) class. I had to mangle things a bit just to get it to compile.

这篇关于HtmlUnit和XPath:DOMNode.getByXPath仅适用于HtmlPage吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！