如何加载HTML的全部内容-Jsoup [英] How to Load Entire Contents of HTML - Jsoup

问题描述

我试图使用jsoup下载html表行，但它仅解析部分html内容.我也尝试使用下面的代码来加载完整的html内容，但是不起作用.任何建议，将不胜感激.

I was trying to download html table rows using jsoup but it parsing only partial html contents. I tried with below code also for loading full html contents but doesn't work. any suggestion would be appreciated.

public class AmfiDaily {
    public static void main(String[] args) {
        AmfiDaily amfiDaily = new AmfiDaily();

        amfiDaily.extractAmfiTable("https://www.amfiindia.com/intermediary/other-data/transaction-in-debt-and-money-market-securities");
    }

    public  void extractAmfiTable(String url){
        Document doc;

        try {
            FileWriter writer = new FileWriter("D:\\FTRACK\\Amfi Report " + java.time.LocalDate.now() + ".csv");
            Document document = Jsoup.connect(url)
                    .userAgent("Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0")
                    .maxBodySize(0)
                    .timeout(100000*5)
                    .get();

            Elements rows = document.select("tr");  

                 for (Element row : rows) {              

                Elements cells1 = row.select("td");                   

                for (Element cell : cells1) {

                    if (cell.text().contains(",")) {

                        writer.write(cell.text().concat(","));

                    }
                    else
                    {
                        writer.write(cell.text().concat(","));
                    }                       

                }                   

                writer.write("\n");                   
                 }
            writer.close();
        } catch (IOException e) {
            e.getStackTrace();
        }
    }
}

推荐答案

禁用JavaScript以准确查看Jsoup看到的内容.页面的一部分已加载AJAX，因此Jsoup无法访问它.但是，有一种简单的方法可以检查其他数据的来源.

Disable JavaScript to see exactly what Jsoup sees. Part of the page is loaded with AJAX so Jsoup is not able to reach it. But there's an easy way to check where the additional data comes from.

您可以使用浏览器调试器检查网络"选项卡，并查看请求和响应.

You can use your browsers debugger to check Network tab and take a look at the requests and responses.

您可以看到该表是从以下URL下载的: https://www.amfiindia.com/modules/LoadModules/MoneyMarketSecurities

You can see that table is downloaded from this URL: https://www.amfiindia.com/modules/LoadModules/MoneyMarketSecurities

您可以直接使用此URL来获取所需的数据.

You can use directly this URL to get the data you need.

要克服Jsoup的限制并立即加载整个HTML，您应该使用Selenium Webdriver，例如此处: https://stackoverflow.com/a/54510107/9889778

To overcome Jsoup's limitation and load whole HTML at once you should use Selenium webdriver, example here: https://stackoverflow.com/a/54510107/9889778

这篇关于如何加载HTML的全部内容-Jsoup的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！