如何查找网站上的所有链接/页面 [英] How to find all links / pages on a website

问题描述

是否可以在任何给定的网站上找到所有页面和链接？我想输入一个URL，并从该网站生成所有链接的目录树？

Is it possible to find all the pages and links on ANY given website? I'd like to enter a URL and produce a directory tree of all links from that site?

我看过HTTrack，但下载整个网站，我只是需要目录树。

I've looked at HTTrack but that downloads the whole site and I simply need the directory tree.

推荐答案

查看 linkchecker - 将抓取网站（同时遵守 robots.txt ）并生成报告。从那里，您可以编写创建目录树的解决方案。

Check out linkchecker—it will crawl the site (while obeying robots.txt) and generate a report. From there, you can script up a solution for creating the directory tree.

这篇关于如何查找网站上的所有链接/页面的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！