带有Selenium-webdriver的AWS Lambda Node.js 10.x运行时错误 [英] AWS Lambda Node.js 10.x Runtime error with selenium-webdriver

问题描述

几天前，我们收到了有关"Lambda操作通知"的通知，将我们的Node.js 8.10运行时更新为Node.js 10.x运行时.

A few days back we received a notification regarding 'Lambda operational notification' to update our Node.js 8.10 runtime to Node.js 10.x runtime.

为响应此通知，我们在开发系统中安装了Node.js版本v10.16.3，并测试了我们现有的代码. 我们发现代码在我们的开发系统中运行良好，但是当我们在带有Node.js 10.x运行时的AWS Lambda中测试相同的代码时，会出现以下错误:

In response to this notification, we installed Node.js version v10.16.3 in our development system and tested our existing code. We found the code was running fine in our development system, but when we tested this same code in AWS Lambda with Node.js 10.x runtime we get this following error:

2019-10-28T12:03:31.771Z 8e2472b4-a838-4ede-bc70-a53aa41d9b79 INFO错误:服务器提前终止，状态为127 在earlyTermination.catch.e(/var/task/node_modules/selenium-webdriver/remote/index.js:251:52) 在process._tickCallback(internal/process/next_tick.js:68:7)

2019-10-28T12:03:31.771Z 8e2472b4-a838-4ede-bc70-a53aa41d9b79 INFO Error: Server terminated early with status 127 at earlyTermination.catch.e (/var/task/node_modules/selenium-webdriver/remote/index.js:251:52) at process._tickCallback (internal/process/next_tick.js:68:7)

'aws-sdk'，'selenium-webdriver'npm软件包和google chrome二进制文件是我们项目中唯一使用的依赖项.

'aws-sdk', 'selenium-webdriver' npm packages and google chrome binaries are the only dependencies used in our project.

我们的项目具有以下文件结构.

Our project has the following file structure.

/var/task/
├── index.js
├── lib
│   ├── chrome
│   ├── chromedriver
│   ├── libgconf-2.so.4
│   ├── libORBit-2.so.0
│   └── libosmesa.so
└── node_modules
    ├── selenium-webdriver
    ├── ...

由于此代码不会在我们的开发系统中引发任何错误，因此我们怀疑它与新的运行时有关.

Since this code is not throwing any error in our development system, we suspect it has to do with the new runtime.

我们尝试使用setChromeBinaryPath()设置二进制路径

We tried the setting the binary path using setChromeBinaryPath()

这是我们正在使用的代码.调用build()方法时会发生错误.

This is the code we are using. The error occurs when the build() method is called.

var webdriver = require('selenium-webdriver');
var chrome = require('selenium-webdriver/chrome');
var builder = new webdriver.Builder().forBrowser('chrome');
var chromeOptions = new chrome.Options();
const defaultChromeFlags = [
    '--headless',
    '--disable-gpu',
    '--window-size=1280x1696', // Letter size
    '--no-sandbox',
    '--user-data-dir=/tmp/user-data',
    '--hide-scrollbars',
    '--enable-logging',
    '--log-level=0',
    '--v=99',
    '--single-process',
    '--data-path=/tmp/data-path',
    '--ignore-certificate-errors',
    '--homedir=/tmp',
    '--disk-cache-dir=/tmp/cache-dir'
];

chromeOptions.setChromeBinaryPath("/var/task/lib/chrome");
chromeOptions.addArguments(defaultChromeFlags);
builder.setChromeOptions(chromeOptions);

var driver = await builder.build();

推荐答案

我们最近遇到了完全相同的问题.从Node v8.x升级到AWS Lambda Node v10.x之后，chrome和chromedriver停止工作.简而言之，根本原因是Lambda Node 10.x在Amazon Linux 2上运行，而Lambda Node v8在Amazon Linux上运行.与以前的版本相比，Amazon Linux 2缺少许多软件包，使其更轻便，但同时如果您要设置自定义运行时环境，则很麻烦.在向您提供解决此问题的步骤之前，让我首先重点介绍一些有用的链接，这些链接可以帮助我找到正确的二进制文件集，而这些二进制文件也必须包含在lambda部署程序包中.

We recently faced the exact same issue. After upgrading to AWS Lambda Node v10.x from Node v8.x, chrome and chromedriver stopped working. In short, the root cause is that Lambda Node 10.x runs on Amazon Linux 2 Vs Lambda Node v8 which runs on Amazon Linux. Amazon Linux 2 lacks a number of packages comparing to it's predecessor, making it more lightweight but at the same time a pain in case you want to set up a custom runtime environment. Before I give you the steps to resolve this, let me first highlight a few useful links that helped me find the right set of binaries I had to also include in my lambda deployment package.

请记住！解决此问题的方法是找出Lambda部署程序包中缺少哪些二进制文件并将其添加.

Just remember! The way to resolve this is to figure out which binaries are missing from your Lambda deployment package and add them in.

1. 如何在AWS Lambda部署程序包中使用Amazon Linux本机二进制程序包.当您知道Lambda环境中缺少某些二进制文件时，AWS的此链接将帮助您将其包含在包中.出于我的目的，我使用EC2 Amazon Linux 64位AMI下载并提取了软件包.详细步骤如下... https://aws.amazon.com/premiumsupport/knowledge-center/lambda-linux-binary-package
2. 除了Amazon Linux 2中缺少的二进制文件外，也没有安装字体.该链接将告诉您如何在AWS Lamda上安装字体. Chrome无法在Lambda上运行的原因之一是缺少字体. https://forums.aws.amazon.com/thread.jspa?messageID=776307
3. 这是github上的一个不错的问题线程，它告诉我LD_LIBRARY_PATH环境变量中的路径顺序很重要.这是保存二进制文件所在路径的环境变量. https://github.com/alixaxel/chrome-aws-lambda/issues/37
4. 现在这是改变游戏规则的人.如果没有创建令人惊叹的docker容器lambci来模拟AWS Lambda到尽可能近的程度，我将永远无法弄清楚.在Amazon Linux 2 EC2服务器和AWS Lambda之间尝试了各种方法之后，这最终成为了我的游乐场，在那里我可以非常快速地迭代尝试不同的软件包. https://hub.docker.com/r/lambci/lambda/
5. 在AWS Lambda中运行任意可执行文件.一些有用的链接，如果您想直接在lambda上运行可执行文件并查看其行为.您从selenium-webdriver软件包中看到的错误消息实际上并未显示chrome或chromedriver引发的实际错误.尝试在lambci docker容器中直接运行chrome或chromedriver是我设法进行调试并找出缺少的二进制文件的方式. https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/

1. How to use Amazon Linux native binary packages in an AWS Lambda deployment package. Once you know you are missing some binaries in your Lambda environment, this link from AWS will help you include them into your package. For my purpose I used an EC2 Amazon Linux 64 bit AMI to download the packages and extract them. Detailed steps follow... https://aws.amazon.com/premiumsupport/knowledge-center/lambda-linux-binary-package
2. Besides binaries missing from Amazon Linux 2, there are also no fonts installed. This link will tell you how to install fonts on AWS Lamda. One of the reasons Chrome fails to run on Lambda is the lack of fonts. https://forums.aws.amazon.com/thread.jspa?messageID=776307
3. This is a nice issue thread on github that taught me that the order of paths in the LD_LIBRARY_PATH environment variable matters. This is the environment variable that holds the paths where your binaries are in. https://github.com/alixaxel/chrome-aws-lambda/issues/37
4. Now this is a game changer. Without the amazing docker container lambci created simulating AWS Lambda to as close as it can be, I would have never figured it out. After trying all sorts of things between an Amazon Linux 2 EC2 server and AWS Lambda, this ended up being my playground, where I could iterate trying different packages very quickly. https://hub.docker.com/r/lambci/lambda/
5. Running Arbitrary Executables in AWS Lambda. Some helpful link if you want to run an executable directly on lambda and see how it behaves. The error messages you see from selenium-webdriver package are actually not surfacing the real error that chrome or chromedriver throws. Trying to directly run chrome or chromedriver in the lambci docker container is how I managed to debug this and figure out which binaries were missing. https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/

所以，这是您需要做的:

So, here is what you need to do:

1. 启动Amazon Linux 2 64位服务器.一个t3.micro应该足够了.
2. SSH到计算机并安装rmpdevtools:sudo yum install -y yum-utils rpmdevtools
3. 创建一个临时目录以下载缺少的软件包:
4. cd /tmp
5. mkdir lib
6. cd lib
7. 下载AWS Lambda节点v10.x中缺少的RPM软件包:yumdownloader --resolve GConf2 glibc glib2 libblkid libffi libgcc libmount libsepol libstdc++ libuuid pcre zlib libselinux dbus-glib mozjs17 polkit polkit-pkla-compat libX11 libX11-common libXau libxcb fontconfig expat fontpackages-filesystem freetype stix-fonts gnu-free-sans-fonts fontpackages-filesystem gnu-free-fonts-common nss nspr nss-softokn nss-softokn-freebl nss-util dbus-libs audit-libs bzip2-libs cracklib elfutils-libelf elfutils-libs libattr libcap libcap-ng libcrypt libdb libgcc libgcrypt libgpg-error libsepol lz4 pam systemd-libs xz-libs mesa-libOSMesa-devel mesa-libOSMesa mesa-libglapi sqlite
8. 提取RPM软件包:rpmdev-extract *rpm
9. 创建一些临时位置，以从提取的RPM工件中复制二进制文件:
   • sudo mkdir -p /var/task
   • sudo chown ec2-user:ec2-user /var/task
   • cd /var/task
   • mkdir lib
   • mkdir fonts

1. Launch an Amazon Linux 2 64 bit server. A t3.micro should be enough.
2. SSH to the machine and install rmpdevtools: sudo yum install -y yum-utils rpmdevtools
3. Create a temporary directory for downloading the missing packages:
4. cd /tmp
5. mkdir lib
6. cd lib
7. Download the RPM packages missing in AWS Lambda node v10.x: yumdownloader --resolve GConf2 glibc glib2 libblkid libffi libgcc libmount libsepol libstdc++ libuuid pcre zlib libselinux dbus-glib mozjs17 polkit polkit-pkla-compat libX11 libX11-common libXau libxcb fontconfig expat fontpackages-filesystem freetype stix-fonts gnu-free-sans-fonts fontpackages-filesystem gnu-free-fonts-common nss nspr nss-softokn nss-softokn-freebl nss-util dbus-libs audit-libs bzip2-libs cracklib elfutils-libelf elfutils-libs libattr libcap libcap-ng libcrypt libdb libgcc libgcrypt libgpg-error libsepol lz4 pam systemd-libs xz-libs mesa-libOSMesa-devel mesa-libOSMesa mesa-libglapi sqlite
8. Extract the RPM packages: rpmdev-extract *rpm
9. Create some temporary location for copying the binaries from the extracted RPM artifacts:
   • sudo mkdir -p /var/task
   • sudo chown ec2-user:ec2-user /var/task
   • cd /var/task
   • mkdir lib
   • mkdir fonts

• /bin/cp /tmp/lib/*/usr/lib64/* /var/task/lib
• /bin/cp /tmp/lib/*/lib64/* /var/task/lib
• /bin/cp /tmp/lib/*/usr/share/fonts/*/*.ttf /var/task/fonts

• /bin/cp /tmp/lib/*/usr/lib64/* /var/task/lib
• /bin/cp /tmp/lib/*/lib64/* /var/task/lib
• /bin/cp /tmp/lib/*/usr/share/fonts/*/*.ttf /var/task/fonts

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <dir>/var/task/fonts/</dir>
  <cachedir>/tmp/fonts-cache/</cachedir>
  <config></config>
</fontconfig>

1. 在您的lambda处理程序中添加以下代码段.这将为LD_LIBRARY_PATH环境变量设置正确的包含路径顺序，还将FONTCONFIG_PATH设置为新的/var/task/fonts目录.

process.env.FONTCONFIG_PATH = `${process.env.LAMBDA_TASK_ROOT}/fonts`;
if (process.env.LD_LIBRARY_PATH.startsWith("/var/task/lib:") !== true) {
  process.env.LD_LIBRARY_PATH = [...new Set(["/var/task/lib", ...process.env.LD_LIBRARY_PATH.split(':')])].join(':');
}

1. 下载本地lambci/lambda图像. docker pull lambci/lambda
2. 通过运行如下所示的lambci图像来调试lamda处理程序:

1. Download locally lambci/lambda image. docker pull lambci/lambda
2. Debug your lamda handler by running a lambci image like this:

docker run --rm -v "$THE_LOCAL_DIR_OF_YOUR_UNCOMPRESSED_LAMDA_PACKAGE":/var/task lambci/lambda:nodejs10.x index.handler

1. 重复步骤7到14，直到可以在lambci容器上使用它为止.使用给定的RPM软件包，它应该可以工作，但是如果不能，则可以通过尝试在lambda中启动chrome来在本地调试正在发生的事情，如下所示:

const childProcess = require('child_process');
childProcess.execFileSync(`${process.env.LAMBDA_TASK_ROOT}/lib/chrome`);

这是一个繁琐的过程，但是最终，您要做的就是将更多的二进制文件添加到程序包中，并在处理程序中添加3行代码以更新lib和fonts环境变量.

This is a cumbersome process, but at the end of the day, all you are doing is just adding some more binaries into your package and 3 lines of code in your handler to update the lib and fonts environment variables.

以防万一，在下面添加我们正在使用的chrome标志:

Just in case, adding below as well the chrome flags we are using:

const defaultChromeFlags = [
  "--headless",
  "--disable-gpu",
  "--window-size=1280x1024",
  "--no-sandbox",
  "--user-data-dir=/tmp/user-data",
  "--hide-scrollbars",
  "--enable-logging",
  "--v=99",
  "--single-process",
  "--data-path=/tmp/data-path",
  "--ignore-certificate-errors",
  "--homedir=/tmp",
  "--disk-cache-dir=/tmp/cache-dir"
];

祝你好运！

这篇关于带有Selenium-webdriver的AWS Lambda Node.js 10.x运行时错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！