Java HTML 使用无限滚动解析页面 [英] Java HTML Parsing a Page with Infinite Scroll

问题描述

如果页面具有无限滚动，我如何在 Java 中抓取页面的 HTML?我目前正在以这种方式抓取页面:

How can I grab a page's HTML in java if the page has infinite scroll? I'm currently grabbing a page this way:

URL url = new URL(stringUrl);
URLConnection con = url.openConnection();
InputStream in = con.getInputStream();
String encoding = con.getContentEncoding();
encoding = encoding == null ? "UTF-8" : encoding;
String html = IOUtils.toString(in, encoding);
Document document = Jsoup.parse(html);

但它不会返回与页面无限滚动部分相关的任何内容.如何在 HTML 页面上触发此滚动，以便我的 Jsoup 文档包含此部分?

But it doesn't return any of the content associated with the infinite scroll section of the page. How can I trigger this scrolling on the HTML page so that my Jsoup document contains this section?

推荐答案

无限滚动描述了一种页面不包含内容的技术.一些 JavaScript 代码在浏览器中运行，向服务器发送请求以获取其他内容并将其添加到页面中.当您滚动到可用内容的末尾时，JavaScript 代码会重复该过程:它会发送另一个请求并添加其他内容.

Infinite scroll describes a technique where the page does not contain the content. Some JavaScript code runs in the browser, sends a request to the server for addiional content and adds it to the page. When you scroll towards the end of the available content, the JavaScript code repeats the process: it sends another request and adds the additional content.

因此，您需要一个带有 JavaScript 引擎的 Web 浏览器，该引擎可以运行 JavaScript 代码并生成导致代码加载内容的事件.

Therefore, you need a web browser with a JavaScript engine that can run the JavaScript code and produce the events that cause the code to load content.

这篇关于Java HTML 使用无限滚动解析页面的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！