Selendroid 作为网页抓取工具 [英] Selendroid as a web scraper

问题描述

我打算创建一个 Android 应用程序，该应用程序执行对网站的无头登录，然后在保持登录会话的同时从后续页面中抓取一些内容.

I intend to create an Android application that performs a headless login to a website and then scrape some content from the subsequent page while maintaining the logged-in session.

我第一次在一个普通的 Java 项目中使用了 HtmlUnit 并且它工作得很好.但是后来发现HtmlUnit与Android不兼容.

I first used HtmlUnit in a normal Java project and it worked just fine. But later found that HtmlUnit is not compatible with Android.

然后我通过向登录表单发送 HTTPPOST"请求来尝试 JSoup 库.但是结果页面没有完全加载，因为 JSoup 不支持 JavaScript.

Then I tried JSoup library by sending HTTP "POST" request to the login form. But the resulting page does not load up completely since JSoup won't support JavaScript.

然后有人建议我查看 Selendroid，它实际上是一个 android 测试自动化框架.但我真正需要的是一个同时支持 JavaScript 和 Android 的 Html 解析器.我发现 Selendroid 很难理解，我什至不知道要使用哪个依赖项.

I was then suggested to have a look on Selendroid which actually is an android test automation framework. But what I actually need is an Html parser that supports both JavaScript and Android. I find Selendroid quite difficult to understand which I can't even figure out which dependencies to use.

• selendroid-client
• selendroid-standalone
• selendroid-server

使用 Selenium WebDriver，代码将如下所示简单.但是有人可以给我看一个类似的 Selendroid 代码示例吗?

    WebDriver driver = new FirefoxDriver();
    driver.get("https://mail.google.com/");

    driver.findElement(By.id("email")).sendKeys(myEmail);
    driver.findElement(By.id("pass")).sendKeys(pass);

    // Click on 'Sign In' button
    driver.findElement(By.id("signIn")).click();

还有，

1. 要向 Gradle.Build 文件添加哪些依赖项?
2. 要导入哪些 Selendroid 库?

推荐答案

很遗憾，我没有让 Selendroid 工作.但我找到了一种解决方法，通过仅使用启用了 JavaScript 的 Android 内置 WebView 来抓取动态内容.

Unfortunately I didn't get Selendroid to work. But I find a workaround to scrape dynamic content by using just Android's built in WebView with JavaScript enabled.

mWebView = new WebView();
mWebView.getSettings().setJavaScriptEnabled(true);
mWebView.addJavascriptInterface(new HtmlHandler(), "HtmlHandler");

mWebView.setWebViewClient(new WebViewClient() {
   @Override
   public void onPageFinished(WebView view, String url) {
       super.onPageFinished(view, url);

       if (url == urlToLoad) {
       // Pass html source to the HtmlHandler
       WebView.loadUrl("javascript:HtmlHandler.handleHtml(document.documentElement.outerHTML);");

   }
});

JS 方法 document.documentElement.outerHTML 将检索包含在加载的 url 中的完整 html.然后将检索到的 html 字符串发送到 HtmlHandler 类中的 handleHtml 方法.

The JS method document.documentElement.outerHTML will retrieve the full html contained in the loaded url. Then the retrived html string is sent to handleHtml method in HtmlHandler class.

class HtmlHandler {
        @JavascriptInterface
        @SuppressWarnings("unused")
        public void handleHtml(String html) {
            // scrape the content here

        }
    }

您可以使用 Jsoup 之类的库从 html 字符串中抓取必要的内容.

You may use a library like Jsoup to scrape the necessary content from the html String.

这篇关于Selendroid 作为网页抓取工具的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！