要求npm:处理重定向 [英] Request npm: Handling Redirects

问题描述

我想知道是否有人知道如何使用请求npm来处理来自位网站，部落网站或Twitter的t.co URL的重定向.例如，如果我有一个要使用请求npm"抓取的网页，并且我必须链接到该页面的链接是一个重定向的网址，该网址太短或太短，我该如何处理这些重定向?

I was wondering if anyone knew how to handle redirects with the Request npm from sites such as bitly or tribal or Twitter's t.co URLs. For example, if I have web page that I want to scrape with the Request npm and the link I have to get to that page is a bity or shortened URL that is going to redirect me, how do I handle those redirects?

我发现Request npm的"followRedirect"选项默认情况下设置为true.如果将其设置为false，则可以通过抓取返回的页面来获取该页面将重定向到的下一个链接，但这不是最好的，因为我不知道我将要进行多少次重定向通过.

I found that the Request npm has a "followRedirect" options set to true by default. If I set that to false I can get the next link that the page will redirect me to by scraping that page that is returned, but that isn't the best because I don't know how many redirects I am going to have to go through.

现在我遇到500错误.当我将"followRedirect"设置为true时.当我将"followRedirect"设置为false时，我可以获取每个重定向页面.同样，我不知道我将要经历多少重定向页面.代码如下:

Right now I am getting a 500 error. When I have "followRedirect" set to true. When I have "followRedirect" set to false, I can get each redirect page. Again, I don't know how many redirect pages I will have to go through. Code is below:

var options = {
  followRedirect: false
};

request('http://t.co/gJ74UfmH4i', options, function(err, response, body){
     // when options are set I get the redirect page
     // when options are not set I get a 500
});

推荐答案

首先，您需要使用followAllRedirects:true参数获取最后的重定向URL

At first, you need to get the last redirect url, using followAllRedirects: true parameter

request('http://t.co/gJ74UfmH4i', {
  method: 'HEAD',
  followAllRedirects: true
}, function(err, response, body) {
  var url = response.request.href
}) 

>

第二部分是使用一些类似于浏览器的标题向最终网址发出请求

The second part is making request to final url, with some browser-like headers

request(url, {
  headers: {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.46 Safari/537.36"
  },
}, function(err, response, body) { 
  //here is your body 
})

这篇关于要求npm:处理重定向的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！