如何处理大数据(约 50 000 个对象)的 Vue 2 内存使用情况 [英] How to handle Vue 2 memory usage for large data (~50 000 objects)

问题描述

我正在尝试在 Vue 2 上为大量半复杂对象实现一个表视图.基本上这个想法是从 DB 中收集 50 000 到 100 000 行之间的任何行到 JS 缓存中，然后动态分析使用实时过滤器(文本搜索)构建表格视图.表格中的每一行都是可切换的，这意味着单击该行会将行更改为编辑模式，从而为该特定字段/单元格启用类似 Excel 的编辑.

每个对象都有大约 100-150 个字段/属性，但在表格中的任何给定时刻只显示一定数量的字段(表格列可以实时切换).对于大型数据集，似乎 DB 正在推送大约 10-100mb 的 JSON 数据，这在这个用例中是可以接受的.Renderwise 性能不是问题——过滤器的工作速度足够快，并且只有有限数量的结果被渲染到 DOM.

一切都已经有效，过滤器，针对过滤器列出约 100 行(+再显示 100 个"-机制等)，但是当我将大约 8000 个对象加载到数组中时，我达到了内存限制.这似乎保留了 2 GB 的 RAM，在 Chrome 停止同时运行 JS 代码之后(尽管奇怪的是我没有收到任何警告/错误).

我对行的内存使用情况进行了基准测试，似乎大约 1000 行保留了大约 300MB 的内存.这很可能是 Vue 反应性观察者保留的.

三个问题:

1. 有没有办法切换特定数组列表对象的反应性(通过索引等)，以便数组本身内的对象是不可观察的/不可改变的，除非专门调用使其变为可变的(即，当用户单击行时，启用编辑模式)?
2. 您将如何处理 Vue 的大型数据集，因为反应性似乎限制了内存使用?请不要建议限制后端内的结果"，因为这不是我在这里寻求的解决方案(即使我可能需要创建两部分过滤，一个用于获取较小的初始数据集和一种用于实时过滤).基本上，我试图通过重新思考 Vue 的数据架构来将内存结束"的界限从 8 000 -> 80 000 推开.将数据集存储在 Vue 的数据变量中是反应性的唯一问题吗?
3. 我的一个想法是使用 Object.freeze 或一些类似的方法将items"-dataset 变为不可观察/非反应性，并使用表格来呈现两个数据集:一个用于非反应性数据集，另一个用于呈现非反应性数据集的数据集当前处于编辑模式(单击行时将被推送到editableItems"数据集)...这里有任何建议(任何更简单的方法，以便我能够处理一个数组中的所有内容?)

我在 Angular 1 上做了类似的应用程序，它处理了 50 000 行相当不错，所以我确定它在 Vue 2 中也应该是可行的......应该只是找到一种处理反应性的方法的问题.

解决方案

Edit 20.4.2021 - 两年后，两年更聪明

由于这个问题/答案受到了很多关注，并且在这么多年之后仍然有效，我想提出一些建议.下面的大部分细节仍然​​有效.尽管如此，在处理过滤结果时，我还是会直接将 VueX 与 Lodash(或本机 JS 函数的现代版本)结合使用.复杂的对象.

为了减轻后端的压力，您可以保持简单:获取没有相关模型的普通对象.这意味着您的主要结果只有相关对象的 ID 键.使用 Axios 或类似库通过单独的 AJAX 请求(客户"、项目"、位置")获取所有相关数据，并使用 VueX 将它们存储在他们自己的列表属性中.为每个创建 getter，例如:

projectsById: state =>{返回 _.keyBy(state.projects, id")},

通过这种方式，您可以在需要时使用相关模型来获取标签和/或完整对象并且您的后端不需要多次获取相关数据.状态和 getter 也将在微组件中可用.

基本上:在处理大型数据集时，避免获取完整的模型树(即使 C# EF 或 PHP Laravel 为它们提供了工具).使用原子方法:获取 20 个不同的列表(Axios.all([...])"是你的朋友！)，每个列表都有自己的控制器端点并将结果缓存到 VueX 存储......并玩得开心;)

编辑 12.03.2019 - 此答案末尾的其他提示

我问这个问题已经有一段时间了，我终于优化了我项目的这一部分.我想为遇到这些性能和/或内存问题的人提供一些建议.

Vue 文档从未真正解释过它，但正如 Andrey 指出的，您可以使用组件对象作为自定义对象的数据存储 &对象列表.毕竟，它只是一个普通的 javascript 对象.

优化后我的列表组件设置看起来像这样:

module.exports = {项目: []，混合:[sharedUtils]，数据:函数(){返回 {列: {全部: []等等...大量的数据和方法

items-array 充满了数以千计的复杂对象(大约 80mb 的数据，6mb 压缩)，我将它们作为非反应性处理.事实证明，这比我想象的要少——我没有直接对项目使用 v-for，我已经在使用这种结构，每当用户点击某个过滤器按钮和/或输入字符串时，我就会触发对这个数组的过滤——过滤(如名称).基本上这个processFilters"方法通过无响应的items-array并返回filteredItems，它存储在数据上下文中.因此，当它发生变异时，它会自动变为反应性.

这样，filteredItems 中的所有项目都保持反应性，但当它们被过滤掉时也会失去反应性，从而节省大量内存.高达 1200 mb 缩小到 400 mb，这正是我正在寻找的.聪明！

需要解决的问题很少.由于数据上下文中不存在项目，因此您不能直接在模板中使用它.这意味着，而不是写...

"0 &&EverythingElseIsReady>

...我必须存储 items-array 的长度来分隔数据道具.这也可以通过计算值来解决，但我喜欢保留这些属性.

放弃主数据数组的反应性毕竟不是一件坏事 - 最重要的部分是了解直接针对该基数组中的项目进行的修改永远不会触发对 UI 的任何更改和/或子组件(douh).只要您以具有隐藏数据容器"的方式分隔代码，这就不应该成为这样的问题.它保存了来自后端的所有结果，并且您拥有该大容器的较小(过滤)表示数组.通过使用良好的 REST 架构，您应该已经可以很好地使用非反应性数据存储，只要您记得在将项目保存在非反应性数据存储中后检查是否也已更新到最新版本.

此外，我对数百行中有多少微组件在性能方面的重要性感到困惑.渲染显然受到了打击，但即使我要传递大型道具数千次(因为我有数千个输入单元实例)，它似乎也没有击中内存.这种对象之一是我的全局翻译键/值对对象，有超过 20 000 行翻译字符串......但它仍然无关紧要.这是有道理的，因为 Javascript 使用对象引用，而 Vue Core 似乎被正确编码，所以只要你使用这样的配置对象作为道具，你就只是从数千个对象中引用同一个数据集.

最后，我会说开始对复杂的 CRUD 对象发疯，而不用担心达到内存限制！

非常感谢安德烈·波波夫 (Andrey Popov) 对正确方向的推动！

提示 (12.03.2019)

因为已经有一段时间了，而且我一直在用大型 & 构建 UI复杂的数据集我决定放弃一些简短的想法 &提示.

1. 考虑您如何管理主记录(即人员或产品)与相关记录(子对象/关系对象).尝试限制为子组件注入的数据量，因为您可能会为不同的主记录多次表示相同的子对象.问题是这些对象可能实际上并不是引用对象！

考虑您有包含城市对象的人对象的情况.多个人住在同一个城市，但是当您从后端获取 JSON 数据时，您确定那些重复的城市对象实际上是同一个城市(人之间共享/引用的城市对象)，还是类似对象的多个表示(带有数据完全相同，但在引擎盖下，每个数据都是一个单独的实例/唯一对象).假设您有 50 000 人，每个人都包含相同的子对象/属性城市":{ id: 4, name: Megatown";}，您是否只是获取了 50 000 个单独的城市实例而不是一个?是 person1.city === person2.city ，还是只是看起来一样但仍然是两个不同的对象?

如果您不确定您是引用共享城市对象还是使用数十个类似子对象的实例，您可以简单地在人员列表组件中进行引用.您的人包含 city-id，因此使用单独的 REST 方法 (getCities) 获取城市列表，并在 UI 级别进行配对.这样你就只有一个城市列表，你可以从那个列表中解析出 city 并将其注入到 person 中，从而只引用一个城市.或者，您可以从列表中解析城市并将其作为属性传递给您的个人组件.

还要确保考虑子对象的目的是什么.你需要它是反应式的，还是静态的?为了节省大量内存，您可以只告诉person.city = city"，它将为每个人组件注入，但是如果它需要是反应式的，那么您需要使用 Vue.set -method... 并记住，如果每个城市都需要自己的实例(这样每个人都有类似的城市对象，但每个人的属性都需要可编辑)，那么您需要确保您没有使用引用的对象！因此，您很可能需要克隆 city-object，这会占用浏览器内存.

2. 您的微组件可能包含只读状态和编辑器状态的单独视图状态.这是很常见的.尽管如此，您实际上每次都在创建该微组件的实例，从而对该组件进行数千次初始化.

考虑一下您拥有带有表格和表格行的类似 Excel 的电子表格的情况.每个单元格都包含您自定义的我的输入".-component，它从您的布局中获取只读"属性.如果 UI 处于只读状态，那么您将仅显示该 my-input-component 内的标签部分，否则您将显示具有某些特殊条件的 input-tag(例如日期时间、数字、文本、textarea，选择标签等).现在让我们假设您有 100 行和 20 列，因此您实际上正在初始化 2000 个 my-input-components.现在的问题是——什么可以改进(性能方面)?

好吧，您可以将 readonly-label 从 my-input-component 分离到您的列表视图，这样您要么显示 readonly-version(标签)，要么显示可编辑的 my-input-component.这样你就有了 v-if 条件，它确保那些 2000 个微组件不会被初始化，除非你特别要求初始化它们(由于行或整个布局从只读 -> 可编辑状态移动)... 当 Vue 不需要创建 2000 个组件时，您可能会猜到对浏览器内存方面的影响有多大.

如果您面临页面加载速度非常慢的问题，那可能根本就不是 VUE.检查呈现到您的 HTML 中的 HTML 标记的数量.当您有大量标签时，HTML 的性能相当差.证明这一点的最简单方法之一是将带有 2000 个选项的 select-tag 重复 100 次，或者使用单个 20000 选项 select-tag.与您可能会通过使用大量带有不必要的包装 div 等的微组件来溢出 html 标签数量的方式相同……您拥有的深度和标签越少，浏览器和渲染性能的要求就越低.中央处理器.

尝试通过示例学习良好的 HTML 标签架构.例如，您可以研究如何对 Trello -services 仪表板视图进行编程.这是相当简单和漂亮的半复杂服务的表示，具有最少的子 div.

---

有很多方法可以改进内存处理，但我认为最重要的方法与分离隐藏"和隐藏"相关.来自可见对象的对象，如我的原始答案所述.第二部分是了解差异或实例化与引用对象.三是限制对象间不必要的数据传递.

就我个人而言，我还没有尝试过这个，但是存在一个 Vue-virtual-scroller 组件，它通过简单地作为看似无限量数据的包装器来处理任意数量的数据.查看概念@https://github.com/Akryum/vue-virtual-scroller ，如果它为您解决了问题，请告诉我.

我希望这些指南能为优化您的组件提供一些想法.永远不要放弃希望，总有进步的空间！

I'm trying to implement an table-view for large collections of semi-complex objects on Vue 2. Basically the idea is to collect anywhere between 50 000 to 100 000 rows from DB into JS cache, which is then analyzed dynamically to build table-view with real-time-filters (text-search). Each row within table is toggleable, meaning that clicking the row changes the row to edit-mode, which enables Excel-like editing for that specific field/cell.

Each object has about ~100-150 fields/properties, but only certain amount of 'em are shown at any given moment within table (table columns can be toggled in real-time). For large datasets it seems that DB is pushing about ~10-100mb of JSON-data, which in this use-case is acceptable. Renderwise the performance isn't an issue -- filters work fast enough and only limited amount of results are rendered to DOM.

Everything already works, filters, listing ~100 rows against filters (+ "show 100 more"-mechanism etc), but I hit memory limit when I loaded around 8000 objects into array. This seems to reserve 2 gigabytes of RAM, which after Chrome stops running JS-code all together (even though strangely I'm not getting any kind of warning/error).

I benchmarked memory usage for rows and it seems that ~1000 rows reserves around 300mb of memory. This is most likely reserved by Vue reactivity observers.

Three questions:

1. Is there a way to toggle reactivity for specific array-list objects (by index or such), so that objects within array itself are unobserved/non-mutable unless specifically called to become mutable (ie. when user clicks row, which enables edit-mode)?
2. How would you implement handling of large datasets for Vue, as reactivity seems to bottleneck the memory usage? Please do not suggest "limit the results within backend", because it's not the solution which I'm seeking here (even though I may need to create two-part filtering, one for fetching smaller initial dataset and one for realtime filtering). Basically I'm trying to push boundaries of "end of memory" from 8 000 -> 80 000 by re-thinking the data-architecture with Vue. Is the only problem having dataset storaged within Vue's data-variables as reactive?
3. One idea I have is to turn that "items" -dataset to non-observable/non-reactive with Object.freeze or some similar approach and have table to render two datasets: one for non-reactive and one for those which are currently within edit-mode (which would be pushed to "editableItems" dataset when row is clicked)... any suggestions here (anything simpler, so that I'm able to handle everything within one array?)

I have done similar application on Angular 1, and it handled 50 000 rows quite well, so I'm sure it should be doable within Vue 2 as well... should be just a matter of finding a way on handling reactivity.

解决方案

Edit 20.4.2021 - Two years later, two years wiser

As this question/answer has gotten lots of attention and is still valid after all the years I wanted to throw few pointers. Most of the details underneath are is still valid. Still, I would direct towards using VueX with Lodash (or modern version of native JS functions) when dealing with filtered results & complex objects.

In order to ease the stress of your backend you can keep things simple: Fetch plain objects without related models. This means that your main results have only ID-keys to related objects. Use Axios or similar library to fetch all the related data with separate AJAX-requests ("customers", "projects", "locations") and use VueX to store 'em in their own list-properties. Create getters for each, such as:

projectsById: state => {
    return _.keyBy(state.projects, "id")
},

This way you can use related models for fetching labels and/or full objects when required and your backend doesn't need to fetch related data more than once. States and getters will be available within micro-components as well.

Basically: Avoid fetching full model-trees (even though C# EF or PHP Laravel provide tools for 'em) when dealing with large datasets. Use atomic approach: Fetch 20 different lists ("Axios.all([...])" is your friend!), each with own controller endpoint and cache the results to VueX store... And have fun ;)

Edit 12.03.2019 - additional tips at the end of this answer

It's been a while since I asked this question and I finally got to optimize this part of my project. I'd like to give few pointers for anyone having these performance and/or memory-issues.

Vue documentation never really explained it, but as Andrey pointed out you CAN use the component-object as an data-storage for your custom objects & object-lists. After all, it's just an normal javascript-object.

After optimization my list component setup looks somewhat like this:

module.exports = {
    items: [],
    mixins: [sharedUtils],
    data: function() {
        return {
            columns: {
                all: []
    etc... Lot's of data & methods

The items-array is filled with thousands of complex objects (about 80mb of data, 6mb compressed) which I'm handling as non-reactive. This proved to be less of an issue than I would have thought -- Instead of using v-for directly against items I was already using structure in which I triggered filtering of this array whenever user clicked some filter-button and/or inputted string-filtering (such as name). Basically this "processFilters"-method goes through non-responsive items-array and returns filteredItems, which is stored in data-context. Thus it automatically becomes reactive as it's mutated.

<tr v-for="item in filteredItems" 

This way all the items within filteredItems stay reactive, but also lose reactivity when they are filtered out, thus saving bunch-load of memory. Whopping 1200mb shrunk to 400mb, which was exactly what I was looking for. Clever!

There are few issues which need to be addressed. Since items doesn't exist in data-context you cannot use it directly within template. This means that instead of writing...

<div v-if="items.length > 0 && everythingElseIsReady">

... I had to store length of items-array to separate data prop. This could have been fixed with computed value as well, but I like to keep those properties existing.

Giving up the reactivity of your main data-array isn't such a bad thing after all - The most important part is to understand that modifications which are made directly against items within that base-array are never triggering any changes to UI and/or sub-components (douh). This shouldn't be such an issue as long as you separate your code in such a way that you have "hidden data container" which holds all the results from backend, and you have smaller (filtered) presentation array of that large container. By using good REST-architecture you should already be good to go with non-reactive data-storage, as long as you remember to check that after saving the item within non-reactive data storage has also been updated to latest revision.

Additionally I was baffled by how little it matters performance-wise how many micro-components there are against hundreds of rows. The render takes a hit obviously, but even if I were to pass large props thousands of times (as I have thousands of instances of input-cells) it didn't seem to hit the memory. One of this kind of objects is my global translations-key/value-pair object, having over 20 000 lines of translated strings... but it still didn't matter. This makes sense, as Javascript uses object-references and Vue Core seems to be properly coded, so as long as you use such configuration objects as props you are simply referring from thousands of objects to the same data-set.

Finally, I'd say start going crazy with complex CRUD objects without fear of hitting memory limit!

Huge thanks for Andrey Popov for giving nudge towards right direction!

Tips (12.03.2019)

Since it's been a while and as I have continued building UI's with large & complex datasets I decided to drop few short ideas & tips.

1. Consider how you manage your master-records (ie. persons or products) vs related records (sub-objects / relational objects). Try to limit the amount of data injected for subcomponents, as you might be representing the same sub-object multiple times for different master-records. The problem is that it's possible that these objects are not actually reference-objects!

Consider situation where you have person-object, which contains city-object. Multiple persons live in the same city, but when you fetch JSON-data from backend are you sure are those duplicated city-objects actually one and same city (shared/referenced city-object between persons), or multiple representations of similar object (with data being exactly same, but under the hood each one being an individual instance / unique object). Let's say that you have 50 000 persons, each one containing the same sub-object/property "city": { id: 4, name: "Megatown" }, did you just fetch 50 000 individual city instances instead of just one? Is person1.city === person2.city , or do they just look the same and still be two different objects?

If you are unsure whether you are refering to shared city-object or using dozens of instances of similar sub-objects you could simply do there referencing inside your person-list-component. Your person contains city-id, so fetch list of cities with separate REST-method (getCities), and do the pairing on UI-level. This way you have only one list of cities, and you could resolve city from that that list and inject it to person, thus making reference to only one city. Alternatively you could resolve the city from list and pass it as an property to your person-component.

Also make sure to consider what is the purpose of the sub-object. Do you need it to be reactive, or is it static? In order to save bunch of memory you could just tell "person.city = city", which will be injected for each and every person-component, but if it needs to be reactive then you need to use Vue.set -method... and remember that if each city needs to be own instance (so that each person has similar city-object, but properties need to be editable per person) then you need to make sure that you are not using referred object! Thus you most likely need to clone the city-object, which will eat up browsers memory.

2. Your micro-component might contain separate view-states for both read-only-state and editor-state. This is quite common. Still, you are actually creating instance of that micro-component every time when, thus initializing that component thousands of times.

Think of situation where you have Excel-like spreadsheet with table and table-rows. Each cell contains your custom "my-input" -component, which takes "readonly"-property from your layout. If the UI is on the readonly-state then you are displaying only the label part inside that my-input-component, but otherwise you are displaying input-tag with some special conditions (such as having different input for datetime, number, text, textarea, select-tag etc). Now let's assume you have 100 rows with 20 columns, so you are actually initializing 2000 my-input-components. Now the question is -- what could be improved (performance-wise)?

Well, you could separate readonly-label from my-input-component to your list-view, so that you either display readonly-version (label) OR you you display the editable my-input-component. This way you have v-if condition, which makes sure that those 2000 micro-components won't be initialized unless you have specifically requested to initialize 'em (due either row or whole layout moving from readonly -> editable -state)... You probably guess how big the impact is memory-wise for browser, when Vue doesn't need to create 2000 components.

If you are facing that your page loads really slow it might not be VUE at all. Check out the amount of HTML-tags rendered to your HTML. HTML performs rather poorly when you have large amounts of tags. One of the simplest ways to demonstrate this is by repeating select-tag with 2000 options 100 times, or by having a single 20000 option select-tag. The same way you might be overflowing the amount of html-tags by having lots of micro-components with unnecessary wrapping divs etc... The less depth and less tags you have, the less rendering performance is required from browser & CPU.

Try to learn good HTML-tag architecture via examples. For an example you could study how Trello -services dashboard-view has been programmed. It's quite simple and beautiful representation of rather semi-complex service, with minimal amount of sub-divs.

---

There are many ways to improve memory handling, but I'd say that most important ones relate to separating "hidden" objects from visible objects, as described on my original answer. Second part is understanding the difference or instanced vs referenced objects. Third is to limit the amount of unnecessary data-passing between objects.

Personally I haven't tried this, but there exists a Vue-virtual-scroller component which handles any amount of data by simply being a wrapper for seemingly infinite amounts of data. Check out the concept @ https://github.com/Akryum/vue-virtual-scroller , and let me know if it solved the problem for you.

I hope these guidelines give some ideas for optimizing your components. Never give up the hope, there is always room for improvement!

这篇关于如何处理大数据(约 50 000 个对象)的 Vue 2 内存使用情况的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！