模拟C#中的无限滚动以获取页面的完整html [英] simulate infinite scrolling in c# to get full html of a page

问题描述

许多网站都使用这种(imo)令人讨厌的无限滚动"样式. 例如tumblr，twitter，9gag等网站.

There are lots of sites that use this (imo) annoying "infinite scrolling" style. Examples of this are sites like tumblr, twitter, 9gag, etc..

我最近尝试使用HtmlAgilityPack通过编程从这些网站上抓取一些照片. 像这样:

I recently tried to scrape some pics off of these sites programatically with HtmlAgilityPack. like this:

HtmlWeb web = new HtmlWeb();  
HtmlDocument doc = web.Load(url);
var primary = doc.DocumentNode.SelectNodes("//img[@class='badge-item-img']");
var picstring = primary.Select(r => r.GetAttributeValue("src", null)).FirstOrDefault();

这很好，但是当我尝试从某些站点加载HTML时，我注意到我只得到了少量的内容(比如说前10个帖子"或图片"或其他内容). ) 这让我想知道是否有可能在c#中模拟页面的向下滚动到底部".

This works fine, but when I tried to load in the HTML from certain sites, I noticed that I only got back a small amount of content (lets say the first 10 "posts" or "pictures", or whatever..) This made me wonder if it would be possible to simulate the "scrolling down to the bottom" of the page in c#.

这不仅是我以编程方式加载html的情况，当我仅访问tumblr之类的站点，并且检查firebug或只是查看源代码"时，我希望所有内容都在某个地方，但是很多似乎是用javascript隐藏/插入的. HTML源中仅显示屏幕上实际可见的内容.

This isn't just the case when I load the html programatically, when I simply go to sites like tumblr, and I check firebug or just "view source", I expected that all the content would be in there somewhere, but alot of it seems to be hidden/inserted with javascript. Only the content that is actually visible on my screen is present in the HTML source.

所以我的问题是:是否可以模拟无限向下滚动到页面并使用c#加载HTML(最好是使用HTML)?

(我知道我可以使用tumblr和twitter的API，但我只是想与HtmlAgilityPack一起玩一些有趣的黑客东西)

(I know that I can use API's for tumblr and twitter, but i'm just trying to have some fun hacking stuff together with HtmlAgilityPack)

推荐答案

除了嵌入Web浏览器(通常在无头环境中通常无法使用)之外，无法一次性可靠地对所有此类网站执行此操作

There is no way to reliably do this for all such websites in one shot, short of embedding a web browser (which typically won't work in headless environments).

您应该考虑做的是查看站点的JavaScript，以便了解在用户向下滚动时使用了哪些AJAX查询来获取内容.

What you should consider doing instead is looking at the site's JavaScript in order to see what AJAX queries are used to fetch content as the user scrolls down.

或者，在浏览器中使用网络调试器(例如Chrome附带的调试器).这些调试器通常具有一个网络"窗格，可用于检查该页面执行的AJAX请求.向下滚动查看这些请求，应该会给您足够的信息来编写模拟这些请求的C#代码.

Alternatively, use a web debugger in your browser (such as the one included in Chrome). These debuggers usually have a "network" pane you can use to inspect AJAX requests performed by the page. Looking at these requests as you scroll down should give you enough information to write C# code that simulates those requests.

然后，您将必须将这些请求的响应解析为特定API提供的任何类型的内容，可能是JSON或XML，但几乎肯定不是HTML. (无论如何这可能对您来说更好，因为它可以节省您解析出面向显示的HTML的麻烦，而AJAX API将为您提供易于使用的数据对象.)

You will then have to parse the response from those requests as whatever type of content that particular API delivers, which will probably be JSON or XML, but almost certainly not HTML. (This may be better for you anyway, since it will save you having to parse out display-oriented HTML, whereas the AJAX API will give you data objects that should be much easier to use.)

这篇关于模拟C#中的无限滚动以获取页面的完整html的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！