使用 wget 从谷歌学者搜索结果中下载所有 pdf 文件 [英] Downloading all pdf files from google scholar search results using wget
问题描述
我想编写一个简单的网络蜘蛛，或者只是使用 wget 从谷歌学者下载 pdf 结果.这实际上是一种获取论文进行研究的好方法.
I'd like to write a simple web spider or just use wget to download pdf results from google scholar. That would actually be quite a spiffy way to get papers for research.
我已阅读有关 stackoverflow 的以下页面:
I have read the following pages on stackoverflow:
如何使用 wget 从网站下载所有文件(但不是 HTML)?
最后一页可能是最鼓舞人心的.我确实按照 this 上的建议尝试使用 wget.
The last page is probably the most inspirational of all. I did try using wget as suggested on this.
我的谷歌学者搜索结果页面是 因此但没有下载任何内容.
My google scholar search result page is thus but nothing was downloaded.
鉴于我对 webspider 的理解水平很低，我应该怎么做才能使这成为可能?我确实意识到编写蜘蛛程序可能非常复杂，并且是我可能不想承担的项目.如果可以使用 wget，那就太棒了.
Given that my level of understanding of webspiders is minimal, what should I do to make this possible? I do realize that writing a spider is perhaps very involved and is a project I may not want to undertake. If it is possible using wget, that would be absolutely awesome.
推荐答案
wget -e robots=off -H --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" -r -l 1 -nd -A pdf http://scholar.google.com/scholar?q=filetype%3Apdf+liquid+films&btnG=&hl=en&as_sdt=0%2C23
注意事项:
- 在搜索查询中使用 filetype:pdf
- 一级递归
- -仅接受 pdf 的 pdf
- -H 跨主机
- -e robots=off 并使用 --user-agent 将确保最佳结果.Google 学术搜索拒绝空白用户代理，而 pdf 存储库可能会禁止使用机器人.
当然限制是这只会点击结果的第一页.你可以扩大递归的深度，但这会变得疯狂并且永远持续下去.我建议使用诸如 Beautiful Soup 和 wget 子进程的组合，以便您可以解析和遍历搜索结果战略上.
The limitation of course is that this will only hit the first page of results. You could expand the depth of recursion, but this will run wild and take forever. I would recommend using a combination of something like Beautiful Soup and wget subprocesses, so that you can parse and traverse the search results strategically.
这篇关于使用 wget 从谷歌学者搜索结果中下载所有 pdf 文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
