搜索引擎如何获取未链接的页面? [英] How do search engines obtain unlinked pages?

问题描述

我注意到很多 Dropbox 页面都被 Google、Bing 等编入索引，并且想知道这些搜索引擎如何获得这样的链接:

https://dl.dropboxusercontent.com/s/85cdji4d5pl5qym/37-71.pdf

https://dl.dropboxusercontent.com/u/11421929/larin2014.pdf

鉴于dl.dropboxusercontent.com 上没有链接可跟踪，路径结构也不是那么容易猜测，搜索引擎怎么可能获得这样的链接?

一种解决方案可能是将其发布在论坛上并被搜索引擎收录，但我查找了很多链接并检查了反向链接，但没有成功.我还注意到 Bing 和 Yahoo 显示的结果比 Google 多得多，这意味着 Bing 在获取这些链接方面做得更好，这对我来说似乎不太可能.

解决方案

即使文档真的没有链接(在他们的网站上没有链接，在其他人的网站上没有链接，没有 站点地图，没有来自在文档中链接的站点的 Referer 日志等)，搜索引擎仍然可以找到链接.

两种方式是:

• 有人可以将 URL 提交给搜索引擎(无论是通过公共工具，还是通过网站的网站管理员帐户).

• 搜索引擎可以获取某些用户在其浏览器中访问的所有 URL.例如，当用户安装了来自该搜索引擎的工具栏时，就会发生这种情况.Bing 就是这种情况，请参阅我的 Webmasters SE 上的相关回答:

  <块引用><块引用>

  Microsoft 已确认他们确实发现并索引了通过用户在安装了 Bing 工具栏的情况下上网找到的 URL.

当然，可能还有更多方法.

I noticed that quite a lot Dropbox pages are indexed by Google, Bing, etc. and was wondering how these search engines obtain for instance links like these:

https://dl.dropboxusercontent.com/s/85cdji4d5pl5qym/37-71.pdf

https://dl.dropboxusercontent.com/u/11421929/larin2014.pdf

Given that there are no links on dl.dropboxusercontent.com to follow and the path structure is not that easy to guess, how is it possible that a search engine obtains such a link?

One solution might be that it was posted on a forum and picked up by the search engine but I looked up quite a lot of the links and checked the backlinks without success. I also noticed that Bing and Yahoo show a considerable amount of more results than Google which would mean that Bing does a better job in picking up these links which seems unlikely to me.

解决方案

Even if the document is really unlinked (no link on their site, no link on someone other’s site, no sitemap, no Referer log from a site that gets linked in the document, etc.), it’s still possible for search engines to find the link.

Two ways are:

• Someone could submit the URL to a search engine (whether via a public tool, or via the site’s webmaster account).

• The search engine could get all URLs that certain users visit in their browsers. This could, for example, happen when the user has installed a toolbar from that search engine. This is the case with Bing, see my related answer on Webmasters SE:

  Microsoft has confirmed that they do discover and index URLs that they find through users surfing the Internet with the Bing Toolbar installed.

And there might be more ways, of course.

这篇关于搜索引擎如何获取未链接的页面?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！