Puppeteer Crawler - 错误：net :: ERR_TUNNEL_CONNECTION_FAILED [英] Puppeteer Crawler - Error: net::ERR_TUNNEL_CONNECTION_FAILED

问题描述

目前我让我的Puppeteer在Heroku上使用代理运行。本地代理中继工作完全正常。但是我收到错误错误：net :: ERR_TUNNEL_CONNECTION_FAILED。我已经在Heroku配置变量中设置了所有.env信息，因此它们都可用。

Currently I have my Puppeteer running with a Proxy on Heroku. Locally the proxy relay works totally fine. I however get the error Error: net::ERR_TUNNEL_CONNECTION_FAILED. I've set all .env info in the Heroku config vars so they are all available.

我知道如何修复此错误并解决问题吗？

Any idea how I can fix this error and resolve the issue?

我目前有

 const browser = await puppeteer.launch({
      args: [
      "--proxy-server=https=myproxy:myproxyport",
      "--no-sandbox",
      '--disable-gpu',
      "--disable-setuid-sandbox",
      ],
      timeout: 0,
      headless: true,
    });

推荐答案

page.authentication

代理服务器参数的正确格式是，

page.authentication

The correct format for proxy-server argument is,

--proxy-server=HOSTNAME:PORT

如果是HTTPS代理，您可以使用页面传递用户名和密码。在进行导航之前验证，

If it's HTTPS proxy, you can pass the username and password using page.authenticate before even doing a navigation,

page.authenticate({username:'user', password:'password'});

完整代码如下所示，

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
        headless:false,
        ignoreHTTPSErrors:true,
        args: ['--no-sandbox','--proxy-server=HOSTNAME:PORT']
  });
  const page = await browser.newPage();

  // Authenticate Here 
  await page.authenticate({username:user, password:password});
  await page.goto('https://www.example.com/');
})();

代理链

如果以某种方式进行身份验证使用上述方法不起作用，您可能希望在其他地方处理身份验证。

Proxy Chain

If somehow the authentication does not work using above method, you might want to handle the authentication somewhere else.

有多个包要做，一个是代理链，有了这个，你可以拿一个代理，并将其用作新的代理服务器。

There are multiple packages to do that, one is proxy-chain, with this, you can take one proxy, and use it as new proxy server.

proxyChain.anonymizeProxy（proxyUrl）将使用一个带有用户名和密码的代理，创建一个可以在脚本上使用的新代理。

The proxyChain.anonymizeProxy(proxyUrl) will take one proxy with username and password, create one new proxy which you can use on your script.

const puppeteer = require('puppeteer');
const proxyChain = require('proxy-chain');

(async() => {
    const oldProxyUrl = 'http://username:password@hostname:8000';
    const newProxyUrl = await proxyChain.anonymizeProxy(oldProxyUrl);

    // Prints something like "http://127.0.0.1:12345"
    console.log(newProxyUrl);

    const browser = await puppeteer.launch({
        args: [`--proxy-server=${newProxyUrl}`],
    });

    // Do your magic here...
    const page = await browser.newPage();
    await page.goto('https://www.example.com');
})();

这篇关于Puppeteer Crawler - 错误：net :: ERR_TUNNEL_CONNECTION_FAILED的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！