限制Chrome无头CPU和内存使用量 [英] Limit chrome headless CPU and memory usage

问题描述

我正在使用硒通过以下命令无头运行chrome:

I am using selenium to run chrome headless with the following command:

system "LC_ALL=C google-chrome --headless --enable-logging --hide-scrollbars --remote-debugging-port=#{debug_port} --remote-debugging-address=0.0.0.0 --disable-gpu --no-sandbox --ignore-certificate-errors &"

但是，似乎无头chrome消耗了过多的内存和cpu，有人知道我们如何限制无头chrome的CPU/内存使用率?或者，如果有一些解决方法.

However it appears that chrome headless is consuming too much memory and cpu,anyone know how we can limit CPU/Memory usage of chrome headless? Or if there is some workaround.

谢谢.

推荐答案

关于不可预测的 CPU 和内存消耗的讨论很多. > Chrome Headless 会话.

There had been a lot of discussion going around about the unpredictable CPU and Memory Consumption by Chrome Headless sessions.

根据讨论情况建立无头的最低限度cpu + mem使用情况 CPU +内存使用情况可以通过以下方式优化:

As per the discussion Building headless for minimum cpu+mem usage the CPU + Memory usage can be optimized by:

• 使用自定义代理或C ++ ProtocolHandlers，您可以返回存根1x1像素的图像，甚至完全阻止它们.
• Chromium小组正在努力通过以下方式添加程序控制产生帧时.目前，无头chrome仍在尝试以 60 fps 进行渲染，这非常浪费.许多页面的确需要几帧(也许 10-20 fps )才能正确呈现(由于使用了requestAnimationFrame和animation triggers)，但是我们希望可以节省很多CPU在这里.
• MemoryInfra 应该可以帮助您确定哪个组件是设置中最大的内存消耗者.

• 用法可以是:

• Using either a custom proxy or C++ ProtocolHandlers you could return stub 1x1 pixel images or even block them entirely.
• Chromium Team is working on adding a programmatic control over when frames are produced. Currently headless chrome is still trying to render at 60 fps which is rather wasteful. Many pages do need a few frames (maybe 10-20 fps) to render properly (due to usage of requestAnimationFrame and animation triggers) but we expect there are a lot of CPU savings to be had here.
• MemoryInfra should help you determine which component is the biggest consumer of memory in your setup.

• An usage can be:

$ headless_shell --remote-debugging-port=9222 --trace-startup=*,disabled-by-default-memory-infra http://www.chromium.org

铬总是将使用尽可能多的资源.如果要有效地限制其利用率，则应考虑使用 cgroups

上面提到的几点是在生产环境中运行无头浏览器时要适应的一些常见最佳实践:

Having said the above mentioned points here are some of the common best practices to adapt when running headless browsers in a production environment:

图:Headless Chrome的资源浪费情况

• 不要运行无头浏览器:

  对于所有帐户，如果可能的话，请不要运行无头的浏览器.无头浏览器是不可预测的且饥饿的.使用浏览器几乎可以完成的所有工作(用于插值和运行JavaScript的工作)都可以通过简单的Linux工具完成.有一些库提供了优雅的Node API，可以通过 HTTP请求和抓取(如果您是最终目标)来获取数据.

  By all accounts, if at all possible, just don't run a headless browser. Headless browsers are un-predictable and hungry. Almost everything you can do with a browser (save for interpolating and running JavaScript) can be done with simple Linux tools. There are libraries those offer elegant Node API's for fetching data via HTTP requests and scraping if that's your end-goal.

  不需要时无需运行无头浏览器:

  有些用户试图保持浏览器处于打开状态，即使在不使用时也是如此，以使浏览器始终可用于连接.尽管这可能是帮助加快会话启动的好策略，但它只会在几个小时后陷入困境.这在很大程度上是因为浏览器喜欢缓存内容并缓慢占用更多内存.每当您不积极使用浏览器时，请关闭它！

  There are users those attempt to keep the browser open, even when not in use, so that it's always available for connections. While this might be a good strategy to help expedite session launch it'll only end in misery after a few hours. This is largely because browsers like to cache stuff and slowly eat more memory. Any time you're not actively using the browser, close it!

  与浏览器而不是页面并行:

  我们仅应在绝对必要时运行一个会话，下一个最佳实践是在每个浏览器中仅运行一个会话.虽然实际上可以通过并行处理页面来节省一些开销，但是如果一个页面崩溃了，它可能会导致整个浏览器崩溃.那样，再加上每个页面都不能保证完全干净(Cookie和存储可能会泄漏).

  We should only run one when absolutely necessary, the next best-practice is to run only one session through each browser. While you actually might save some overhead by parallelizing work through pages, if one page crashes it can bring down the entire browser with it. That, plus each page isn't guaranteed to be totally clean (cookies and storage might bleed-through).

  page.waitForNavigation :

  page.waitForNavigation:

  观察到的最常见问题之一是触发页面加载的操作以及脚本执行的突然丢失.这是因为触发pageload的动作通常会导致后续工作被吞噬.为了解决此问题，通常必须调用page-loading-action，并立即等待下一次页面加载.

  One of the most common issues observed are the actions that trigger a pageload, and the sudden loss of your scripts execution. This is because actions that trigger a pageload can often cause subsequent work to get swallowed. In order to get around this issue, you will generally have to invoke the page-loading-action and immediately wait for the next pageload.

  使用docker包含所有内容:

  Chrome需要大量依赖才能正常运行.即使所有这些都完成了，您仍然需要担心诸如字体和幻影处理之类的事情，因此使用某种容器来容纳它是理想的. Docker几乎是为此任务定制的，因为您可以限制可用资源的数量并将其沙箱化.自己创建自己的 Dockerfile .

  Chrome takes a lot of dependencies to get running properly. Even after all of that's complete then there's things like fonts and phantom-processes you have to worry about so it's ideal to use some sort of container to contain it. Docker is almost custom-built for this task as you can limit the amount resources available and sandbox it. Create your own Dockerfile yourself.

  为避免陷入僵尸进程(通常在Chrome中发生)，您需要使用 dumb-init 之类的东西来正确启动.

  And to avoid running into zombie processes (which commonly happen with Chrome), you'll want to use something like dumb-init to properly start-up.

  两个不同的运行时:

  可能有两个 JavaScript运行时(节点和浏览器).这对于共享性而言非常有用，但是却以混乱为代价，因为某些页面方法将要求您显式传递引用(而不是使用闭包或提升).

  There can be two JavaScript runtimes going on (Node and the browser). This is great for the purposes of shareability, but it comes at the cost of confusion since some page methods will require you to explicitly pass in references (versus doing so with closures or hoisting).

  例如，在协议的最下方使用page.evaluate时，实际上是stringifies函数并将其传递给Chrome，因此闭包和提升之类的功能根本不起作用.如果您需要将一些引用或值传递到评估调用中，只需将它们附加为可以正确处理的参数即可.

  As an example, while using page.evaluate deep down in the bowels of the protocol, this literally stringifies the function and passes it into Chrome, so things like closures and hoisting won't work at all. If you need to pass some references or values into an evaluate call, simply append them as arguments which get properly handled.

  参考:运行200万次无头会话的观察

  这篇关于限制Chrome无头CPU和内存使用量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！