Node.js 中的内存泄漏 - 如何分析分配树/根? [英] Memory leaks in Node.js - How to analyze allocation tree/roots?

问题描述

查找内存泄漏是一项非常困难的任务，尤其是在涉及使用许多第三方库的现代 JS 代码时.

例如，我目前正面临 rollup 中的内存泄漏，涉及 babel 和自定义 babel 插件.

我正在探索几种常见的策略来追捕它们:

1. 了解您的运行时及其内存取消分配方案，并遵循有关该方案的最佳做法.

   • 遗憾的是，--heap-prof 记录的日志只包含模式 1 的数据.但是，这种模式不足以回答 OP 的第三个问题:如何找出分配的原因/位置对象仍然挥之不去(即:不再使用后保留")?

     如标签中所述:回答该问题需要第二种模式.

     我不知道有没有隐藏的方法可以更改Node的配置文件模式，但我没有找到.我尝试了一些方法，包括从

     • Constructor 视图列出了所有延迟对象，按构造函数/类型分组.
       • 确保您是按 Shallow Size 或 Retained Size 排序(两者都有说明

         我们看到三个函数在这个对象的挥之不去中发挥作用:

         • 调用 gc 的函数 - 我不知道为什么会这样.可能与GC内部有关.可能是因为 gc 会缓存对某些(如果不是全部)延迟对象的引用.
         • addPressure 函数分配了对象.这也是保留它的引用的来源.
         • test1 函数是我们将对象分配给文件范围的 a 的地方.
           • 这是真正的泄漏！我们可以通过不将其分配给 a 来修复它，或者确保在不再使用 a 后清除它.

         结论

         我希望，这可以帮助您开始寻找和消除内存泄漏的激动人心的旅程.欢迎在下方询问更多信息.

         Finding memory leaks is a very difficult task, especially when it comes to modern JS code that makes use of many third party libraries.

         For example, I am currently facing down a memory leak in rollup, involving babel and a custom babel plugin.

         I am exploring several common strategies to hunting them down:

         1. Understand your runtime, its memory de-allocation scheme, and follow best practices regarding that scheme.
            • This article claims that all modern JS runtime implementations use a Mark-and-sweep garbage collector. One of its major strengths is that it can properly deal with circular references. (The article also links this very outdated workshop paper. Don't pay much attention to it, since it is all about circular references, which should not be an issue anymore.)
            • This article goes in-depth on V8 memory management (NOTE: Node and Chrome are both based on V8).
         2. If you find that memory or GC usage explodes beyond your expectation, analyze your heap memory profile to find out where memory gets allocated.
            • This SO answer explains how to do that in Chrome, but its links are outdated. This is a direct link to the relevant Chrome documentation (as of 2021).
            • For Node, I found a lot of outdated information. Currently, the easiest way to analyze your heap memory profile seems to be using the experimental --heap-prof command line argument (e.g. node --heap-prof node_modules/rollup/dist/bin/rollup -c to analyze a rollup build). Then open it in Chrome Dev Tools, via Memory -> Load.
            • Once analyzed, we can understand where/how most memory was allocated; but one crucial question has not yet been answered:
         3. Given you know how the memory was allocated, how can you find out why/where they are still lingering?
            • Given a mark-sweep GC, the answer to this question can ideally be answered by investigating the "object allocation tree".
            • Maybe, most importantly, we would want to be able to answer the question: "What is the GC root (stack pointer) of a given object?"

         This last question is also my question here: How can we analyze the object allocation tree in Node (or in V8 in general)? How can I find out where the objects that I identified in step (2) are kicking around?

         Often, it is the answer to this question that tells us where to change our code to stop the leakage. (Of course, if your issue is memory churn, then usually, this question is not important.)

         In my example, I know that the memory is occupied by Babel AST nodes and path objects, but I don't know why they linger, that is I don't know where they are stored. If you just run Babel on its own, you can verify that it is not Babel leaking the memory. I am currently trying all kinds of tricks to find out where they are being stored, but still no luck.

         Sadly, so far, I have not found any tools to help with question (3). Even relevant in-depth articles (like this and its slidedeck here) MANUALLY draw up heap allocation steps. Feels like there is no such tool, or am I wrong? If there is no tool, maybe is there a discussion about this somewhere?

         解决方案

         Note that while you don't have to explicitely deallocate memory in JS, memory leaks can still arise. At the same time, Node memory profiling utilities are (almost criminally) underdocumented. Let's find out how to use them.

         Memory Leaks in JS

         Since JS has a GC, memory leaks only have a few possibly causes:

         • You are hanging on to ("retaining") large objects, that are not used anymore, usually inside a variable in file or global scope. This is either accidental, or, part of a simplistic (indefinite) caching scheme:

           let a;
           function f() {
             a = someLargeObject;
           }
           

         • Sometimes objects are lingering in retained closures. E.g.:

           let cb;
           function f() {
             const a = someLargeObject;  // `a` is retained as long as `cb`
             cb = function g() {
               eval('console.log(a)');
             };
           }
           

         You can easily fix such a memory leak by either never storing to, or by manually clearing those variables. The main difficulty is to find these lingering objects.

         Using Chrome Dev Tools to Profile Node Applications

         Firstly, Node.js and Chrome both use the same JS engine: v8. Because of that, it was feasible for the Chrome Dev Tools team to add Node debugging and profiling support. While there are other tools available, Chrome Dev Tools (CDT) are probably more mature (and probably much better funded), which is why we will (for now) focus on how to use Chrome Dev Tools for Node memory profiling and debugging.

         There are two main ways of profiling Node memory using CDT:

         1. Run your app with --heap-prof to generate a heap profile log file. Then load and analyze the log in CDT.
         2. Run your app with --inspect/--inspect-brk flag in order to debug your Node application in CDT. Then just use CDT's Memory tab (documentation here) to your liking.

         Method 1: heap-prof

         Run your app with --heap-prof to generate a heap profile log file. Then load and analyze the log in CDT.

         Steps

         1. Run your application with heap-prof enabled. E.g.: node --heap-prof app.js
            • You can further customize this with other heap-prof-related command line flags.
         2. Look into the working directory (usually the folder from where you are running the application). There is a new file which, by default, is named Heap*.heapprofile.
         3. Open a new tab in Chrome → open CDT → go to Memory tab
         4. At the bottom, press Load → select Heap*.heapprofile
         5. Done. You can now see where memory, still alive at the end of the recording, was allocated.

         Considerations for Method 1

         This step allows you to, first of all, verify a memory leak, and find out what kind of allocations or objects might be causing it.

         Let's look at CDT's memory profiling tool. It has three modes:

         Sadly, the log recorded by --heap-prof only contains data for mode 1. However, this mode is insufficient to answer the OP's third question: How can you find out why/where allocated objects are still lingering (that is: "retained" after not being used anymore)?

         As explained in the tab: Answering that question requires the second mode.

         I don't know if there is a hidden way to change the profile mode for Node, but I have not found it. I tried a few things, including adding from this list of undocumented Node.js CLI flags.

         That is why @jmrk proposed method (2) in his answer:

         Method 2: inspect/inspect-brk

         Run your app with --inspect/--inspect-brk flag in order to debug your Node application in CDT. Then just use CDT's Memory tab (documentation here) to your liking.

         Steps

         1. Run application in debug mode, and halt execution at the beginning: node --inspect-brk app.js
         2. Open chrome://inspect in Chrome.
         3. After a few seconds, your application should show up in the list. Select it.
         4. CDT are launched and you see that execution is halted at the entry point of your application.
         5. Go to the Memory tab, select the 2nd mode and press the "Record" button
         6. Continue execution until the memory leak was recorded. For this, either put down a breakpoint somewhere, or, if the leak persists until the end, just let the app exit naturally.
         7. Go back to the Memory tab and press the "Record" button again to stop recording.
         8. You can now analyze the log (see below).

         Considerations for Method 2

         1. Because you are now running your entire application in debug mode, everything is a lot slower.
         2. Heap Mode 2 generally requires a lot more memory. If memory exceeds your Node default memory limit (about 2gb), it will just crash. Monitor your memory usage, and possibly use something like --max-old-space-size=4096 (or bigger numbers) to double the default. Or, even better, simplify your test case to use less memory and speed up profiling, if possible.
         3. The "Record Allocation Stacks" option shows you the call stack of when any object was allocated. That is similar to the functionality of Profile mode 1. It is not necessary for finding memory leaks. I have not needed it so far, but if you need to map the lingering objects to their allocations, this should help.

         Finding Memory Leaks

         After following the steps of Method 2, you are now looking at all information you need to find your leak.

         Let's look at some basic examples:

         Example 1

         Code

         A simplistic memory leak is examplified in the code below: file-scoped a stores data forever.

         Complete Gist is here.

         let a;
         function test1() {
           const b = [];
           addPressure(N, b);
           a = b;
           gc(); // --expose-gc
         }
         
         test1();
         debugger;
         

         Notes:

         • It is our goal to find "lingering" objects; which are "non-collectable" objects; objects that have been retained even though they are not used anymore. That is why I would usually call gc when profiling. This way we can make sure we get rid of all collectable references, and focus explicitly on the "lingering" objects.
           • You need the expose-gc flag for the gc() call; e.g.: node --inspect-brk --expose-gc app.js

         Memory View

         Once the breakpoint hits, I stop recording and I get this:

         • The Constructor view lists all lingering objects, grouped by constructor/type.
           • Make sure, you are sorting by Shallow Size or Retained Size (both are explained here)
         • We find that string is using up most memory. Let's open that up.
           • Below every Constructor, you find a list of all it's individual objects. The first (biggest) object(s) is/are often times the culprit. Select the first.
         • The Retainers view now shows you where this object is still being retained.
           • Here you want to find the function that retained it for the long term (making it "linger").

         Documentation on the Retainers view is not quite complete. This is how I try to navigate it until it spits out the line of code that I'm looking for:

         • Select an object.
           • (Again, it's usually easiest to work through this list, sorted by size.)
         • Inside the object's tree view entry: open up nested tree view entries.
         • Look for anything refering to a line of code (displayed on the right-hand-side of the first column).
         • Entries labeled with "context" might be more useful than others.

         My findings are shown in this screenshot:

         We see three functions playing a role in this object's lingering:

         • The function that called gc - I'm not sure why this is. Probably related to GC internals. Might be because the gc would cache references to some (if not all) lingering objects.
         • The addPressure function allocated the object. This is also where the reference that retained it came from.
         • The test1 function is where we assigned the object to the file-scoped a.
           • This is the actual leak! We can fix it by either not assigning it to a, or make sure, we clear a after it's not being used anymore.

         Conclusion

         I hope, this helps you get started on your exciting journey to finding and eradicating your memory leaks. Feel free to ask for more information below.

         这篇关于Node.js 中的内存泄漏 - 如何分析分配树/根?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！