无法在 DigitalOcean 上运行而不窒息的多个节点子进程 [英] Unable to Run Multiple Node Child Processes without Choking on DigitalOcean

问题描述

我一直在努力在 DigitalOcean 上运行多个 Puppeteer 实例，但运气不佳.我可以使用 puppeteer-cluster 之类的工具同时运行 ~5，但出于某种原因整个事情只是窒息，几乎没有有用的信息.因此，我切换到生成约 5 个子进程而不需要任何额外的库——只是 Puppeteer 本身.同样的问题.没有有用错误的窒息.

I've been struggling to run multiple instances of Puppeteer on DigitalOcean for quite some time with little luck. I'm able to run ~5 concurrently using tools like puppeteer-cluster, but for some reason the whole thing just chokes with little helpful messaging. So, I switched to spawning ~5 child processes without any additional library -- just Puppeteer itself. Same issue. Chokes with no helpful errors.

我能够在本地很好地运行所有这些作业，但是在我部署后，我遇到了这些问题.所以，我的预感是这是一个资源/性能问题，但我不能肯定地说.

I'm able to run all of these jobs just fine locally, but after I deploy, I hit these walls. So, my hunch is that it's a resource/performance issue, but I can't say for sure.

我正在 Digital Ocean 上运行一个 1GB 和 3CPU 的 Droplet.

I'm running a droplet with 1GB and 3CPUs on Digital Ocean.

基本上，我只是在寻找开始对此类问题进行故障排除的方法.有没有办法可以确定我正在撞到资源墙?我试过 pm2 和 DO 仪表板图表，但我觉得这些都遗漏了很多信息，否则我完全错过了其他东西.

Basically, I'm just looking for ways to start troubleshooting something like this. is there a way I can know for sure that I'm hitting resource walls? I've tried pm2 and the DO dashboard graphs, but I feel like those are all leaving a lot of information out, or else I'm missing something else altogether.

推荐答案

puppeteer-cluster 的作者在这里.您是对的，1 GB 的内存可能不足以运行 5 个浏览器窗口(或选项卡)以及您的操作系统和其他后台任务.

Author of puppeteer-cluster here. You are right, 1 GB of memory is likely not enough for running 5 browser windows (or tabs) in addition to your operating system and maybe even other background tasks.

以下是您应该检查的资源列表:

Here is a list of resources you should check:

• 内存:使用像 htop 在应用程序运行时检查内存使用情况.
• CPU:同样，您可以使用 htop，3 个 vCPU 应该足以满足 5 个窗口的需求.
• 磁盘空间:使用df之类的工具检查磁盘是否有足够的空间.我知道磁盘空间不足的多种情况(例如一些旧内核填充磁盘)，Chrome 至少需要一些空间才能运行.
• 网络吞吐量:很少出现问题，但有时网络没有足够的带宽来支持许多打开的浏览器.使用像 nload 这样的工具来检查网络吞吐量.

• Memory: Use a tool like htop to check your memory usage while your application is running.
• CPU: Again, you can use htop for that, 3 vCPUs should be more than enough for 5 windows.
• Disk space: Use a tool like df to check if there is enough space on the disk. I know of multiple cases in which there was not enough space on the disk (like some old kernels filling the disk), and Chrome needs at least some space to run.
• Network throughput: Rarely the problem, but sometimes the network just does not have the bandwidth to support many open browser. Use a tool like nload to check the network throughput.

要使用 htop 或 nload，请在后台启动脚本(node script.js &)或使用终端多路复用器(例如 tmux).资源问题应该很容易发现.

To use htop or nload, you start your script in the background (node script.js &) or use a terminal multiplexer (like tmux). Resource problems should then be easy to spot.

这篇关于无法在 DigitalOcean 上运行而不窒息的多个节点子进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！