如何处理大数据(约50000个对象)的Vue 2内存使用情况 [英] How to handle Vue 2 memory usage for large data (~50 000 objects)
问题描述
我正在尝试为Vue 2上的大量半复杂对象实现一个表视图.基本上，这个想法是从数据库到JS缓存中收集5万到10万行之间的任何内容，然后对其进行动态分析使用实时过滤器(文本搜索)构建表格视图.表格中的每一行都是可切换的，这意味着单击该行会将行更改为编辑模式，从而可以对该特定字段/单元格进行类似Excel的编辑.
I'm trying to implement an table-view for large collections of semi-complex objects on Vue 2. Basically the idea is to collect anywhere between 50 000 to 100 000 rows from DB into JS cache, which is then analyzed dynamically to build table-view with real-time-filters (text-search). Each row within table is toggleable, meaning that clicking the row changes the row to edit-mode, which enables Excel-like editing for that specific field/cell.
每个对象具有约100-150个字段/属性，但是在表中的任何给定时刻仅显示一定数量的'em(表列可以实时切换).对于大型数据集，似乎DB正在推送约10-100mb的JSON数据，在这种用例中，这是可以接受的.从性能上讲，性能不是问题-过滤器的运行速度足够快，并且仅将有限的结果呈现给DOM.
Each object has about ~100-150 fields/properties, but only certain amount of 'em are shown at any given moment within table (table columns can be toggled in real-time). For large datasets it seems that DB is pushing about ~10-100mb of JSON-data, which in this use-case is acceptable. Renderwise the performance isn't an issue -- filters work fast enough and only limited amount of results are rendered to DOM.
所有东西都可以使用，过滤器，针对过滤器列出约100行(+显示100个以上"机制，等等)，但是当我将大约8000个对象加载到数组中时，我达到了内存限制.这似乎保留了2 GB的RAM，在Chrome停止一起运行所有JS代码之后(即使很奇怪，我没有得到任何警告/错误).
Everything already works, filters, listing ~100 rows against filters (+ "show 100 more"-mechanism etc), but I hit memory limit when I loaded around 8000 objects into array. This seems to reserve 2 gigabytes of RAM, which after Chrome stops running JS-code all together (even though strangely I'm not getting any kind of warning/error).
我对行的内存使用情况进行了基准测试，似乎〜1000行保留了大约300mb的内存.这很可能是Vue反应性观察者保留的.
I benchmarked memory usage for rows and it seems that ~1000 rows reserves around 300mb of memory. This is most likely reserved by Vue reactivity observers.
三个问题:
- 有没有一种方法可以切换特定数组列表对象的反应性(通过索引等)，以便数组本身内的对象是不可观察的/不可改变的，除非被明确地称为可变的(即，当用户单击行时，这是可变的)启用编辑模式)?
- 由于反应性似乎会限制内存使用量，您将如何为Vue实施大型数据集的处理?请不要建议将结果限制在后端之内"，因为这不是我在这里寻求的解决方案(即使我可能需要创建两部分的过滤器，其中一个过滤器用于获取较小的初始数据集，一种用于实时过滤).基本上，我试图通过重新考虑Vue的数据体系结构来将内存末尾"的界限从8 000-> 80 000.将数据集存储在Vue的数据变量中是反应性的唯一问题吗?
- 我的一个想法是使用Object.freeze或类似的方法将项目"-数据集设置为不可观察/非反应性的，并具有用于渲染两个数据集的表:一个用于非反应性的数据集，另一个用于处理那些当前处于编辑模式(单击行时将被推送到"editableItems"数据集)...此处有任何建议(更简单的方法，以便我能够处理一个数组中的所有内容?)
- Is there a way to toggle reactivity for specific array-list objects (by index or such), so that objects within array itself are unobserved/non-mutable unless specifically called to become mutable (ie. when user clicks row, which enables edit-mode)?
- How would you implement handling of large datasets for Vue, as reactivity seems to bottleneck the memory usage? Please do not suggest "limit the results within backend", because it's not the solution which I'm seeking here (even though I may need to create two-part filtering, one for fetching smaller initial dataset and one for realtime filtering). Basically I'm trying to push boundaries of "end of memory" from 8 000 -> 80 000 by re-thinking the data-architecture with Vue. Is the only problem having dataset storaged within Vue's data-variables as reactive?
- One idea I have is to turn that "items" -dataset to non-observable/non-reactive with Object.freeze or some similar approach and have table to render two datasets: one for non-reactive and one for those which are currently within edit-mode (which would be pushed to "editableItems" dataset when row is clicked)... any suggestions here (anything simpler, so that I'm able to handle everything within one array?)
我在Angular 1上做了类似的应用程序，它可以很好地处理5万行，所以我确定它在Vue 2中也应该可行...应该只是找到一种处理反应性的方法
I have done similar application on Angular 1, and it handled 50 000 rows quite well, so I'm sure it should be doable within Vue 2 as well... should be just a matter of finding a way on handling reactivity.
推荐答案
编辑12.03.2019-此答案结尾的其他提示
Edit 12.03.2019 - additional tips at the end of this answer
自从我问了这个问题已经有一段时间了，我终于对我的项目的这一部分进行了优化.对于那些具有这些性能和/或内存问题的人，我只想提供一些指导.
It's been a while since I asked this question and I finally got to optimize this part of my project. I'd like to give few pointers for anyone having these performance and/or memory-issues.
Vue文档从未真正解释过它，但是正如Andrey指出的那样，您可以将组件对象用作自定义对象的数据存储.对象列表.毕竟，这只是一个普通的javascript对象.
Vue documentation never really explained it, but as Andrey pointed out you CAN use the component-object as an data-storage for your custom objects & object-lists. After all, it's just an normal javascript-object.
优化后，我的列表组件设置看起来像这样:
After optimization my list component setup looks somewhat like this:
module.exports = {
items: [],
mixins: [sharedUtils],
data: function() {
return {
columns: {
all: []
etc... Lot's of data & methods
items-array充满了成千上万个复杂对象(大约80mb的数据，压缩后的6mb)，我将其视为非反应性的.事实证明，这比我想象的要少-与直接使用v-for代替项目相比，我已经使用了一种结构，在该结构中，当用户单击某些过滤器按钮和/或输入字符串时，我会触发该数组的过滤.过滤(例如名称).基本上，此"processFilters"方法通过无响应的项目数组并返回已过滤的项目，该项目存储在数据上下文中.因此，它在变异时会自动变得反应灵通.
The items-array is filled with thousands of complex objects (about 80mb of data, 6mb compressed) which I'm handling as non-reactive. This proved to be less of an issue than I would have thought -- Instead of using v-for directly against items I was already using structure in which I triggered filtering of this array whenever user clicked some filter-button and/or inputted string-filtering (such as name). Basically this "processFilters"-method goes through non-responsive items-array and returns filteredItems, which is stored in data-context. Thus it automatically becomes reactive as it's mutated.
<tr v-for="item in filteredItems"
这样，filteredItems中的所有项目都保持反应性，但在滤除它们时也会失去反应性，从而节省了大量的内存.将1200mb缩小到400mb，这正是我想要的.聪明！
This way all the items within filteredItems stay reactive, but also lose reactivity when they are filtered out, thus saving bunch-load of memory. Whopping 1200mb shrunk to 400mb, which was exactly what I was looking for. Clever!
需要解决的问题很少.由于项目不在数据上下文中，因此您不能在模板中直接使用它.这意味着不用写...
There are few issues which need to be addressed. Since items doesn't exist in data-context you cannot use it directly within template. This means that instead of writing...
<div v-if="items.length > 0 && everythingElseIsReady">
...我必须存储项目数组的长度以分隔数据道具.也可以使用计算值解决此问题，但是我想保留这些属性.
... I had to store length of items-array to separate data prop. This could have been fixed with computed value as well, but I like to keep those properties existing.
放弃主数据数组的响应性并不是一件坏事-最重要的部分是要了解直接针对该基本数组中的项目进行的修改永远不会触发对UI的任何更改和/或子组件(douh).只要您以这样的方式分离代码，即您拥有隐藏的数据容器"(其中包含来自后端的所有结果)，并且具有较大的容器(经过过滤的)较小的表示数组，那么这应该不是问题.通过使用良好的REST架构，只要您记得检查将项目保存在非反应性数据存储中后是否也已更新到最新版本，那么使用非反应性数据存储就应该很好.
Giving up the reactivity of your main data-array isn't such a bad thing after all - The most important part is to understand that modifications which are made directly against items within that base-array are never triggering any changes to UI and/or sub-components (douh). This shouldn't be such an issue as long as you separate your code in such a way that you have "hidden data container" which holds all the results from backend, and you have smaller (filtered) presentation array of that large container. By using good REST-architecture you should already be good to go with non-reactive data-storage, as long as you remember to check that after saving the item within non-reactive data storage has also been updated to latest revision.
另外，我对性能影响不大感到困惑，因为在数百行中有多少个微组件.渲染显然受到了打击，但是即使我要传递大型道具数千次(因为我有成千上万个输入单元的实例)，它似乎也没有击中内存.这种对象之一是我的全局翻译键/值对对象，它具有超过2万行已翻译的字符串……但仍然没有关系.这是有道理的，因为Javascript使用对象引用，并且Vue Core似乎已正确编码，因此，只要您使用诸如props之类的配置对象，您就可以简单地将成千上万个对象引用到同一数据集.
Additionally I was baffled by how little it matters performance-wise how many micro-components there are against hundreds of rows. The render takes a hit obviously, but even if I were to pass large props thousands of times (as I have thousands of instances of input-cells) it didn't seem to hit the memory. One of this kind of objects is my global translations-key/value-pair object, having over 20 000 lines of translated strings... but it still didn't matter. This makes sense, as Javascript uses object-references and Vue Core seems to be properly coded, so as long as you use such configuration objects as props you are simply referring from thousands of objects to the same data-set.
最后，我要说的是开始对复杂的CRUD对象发疯，而不必担心达到内存限制！
Finally, I'd say start going crazy with complex CRUD objects without fear of hitting memory limit!
非常感谢Andrey Popov向正确的方向轻推！
Huge thanks for Andrey Popov for giving nudge towards right direction!
提示(12.03.2019)
由于已经有一段时间了，并且随着我继续使用大型&复杂的数据集，我决定放弃一些简短的想法&提示.
Since it's been a while and as I have continued building UI's with large & complex datasets I decided to drop few short ideas & tips.
- 考虑如何管理主记录(即人或产品)与相关记录(子对象/关系对象).尝试限制为子组件注入的数据量，因为对于不同的主记录，您可能多次表示同一子对象.问题在于这些对象实际上可能不是引用对象！
请考虑您有包含城市对象的人对象的情况.多个人居住在同一个城市，但是当您从后端获取JSON数据时，您确定这些重复的城市对象实际上是一个城市和同一城市(人与人之间共享/引用的城市对象)，还是相似对象的多个表示(带有数据完全相同，但实际上每个数据都是一个单独的实例/唯一对象).假设您有5万个人，每个人都包含相同的子对象/属性城市":{id:4，名称:"Megatown"}，您是否仅获取了50000个单独的城市实例而不是一个?是person1.city === person2.city，还是它们看起来一样并且仍然是两个不同的对象?
Consider situation where you have person-object, which contains city-object. Multiple persons live in the same city, but when you fetch JSON-data from backend are you sure are those duplicated city-objects actually one and same city (shared/referenced city-object between persons), or multiple representations of similar object (with data being exactly same, but under the hood each one being an individual instance / unique object). Let's say that you have 50 000 persons, each one containing the same sub-object/property "city": { id: 4, name: "Megatown" }, did you just fetch 50 000 individual city instances instead of just one? Is person1.city === person2.city , or do they just look the same and still be two different objects?
如果不确定是要引用共享的城市对象还是使用数十个类似子对象的实例，则可以在此简单地在人员列表组件内部进行引用.您的人包含city-id，因此请使用单独的REST方法(getCities)获取城市列表，并在UI级别进行配对.这样，您只有一个城市列表，并且可以从该列表中解析城市并将其注入到人中，从而仅引用一个城市.或者，您可以从列表中解析城市并将其作为属性传递给您的人员组件.
If you are unsure whether you are refering to shared city-object or using dozens of instances of similar sub-objects you could simply do there referencing inside your person-list-component. Your person contains city-id, so fetch list of cities with separate REST-method (getCities), and do the pairing on UI-level. This way you have only one list of cities, and you could resolve city from that that list and inject it to person, thus making reference to only one city. Alternatively you could resolve the city from list and pass it as an property to your person-component.
还要确保考虑子对象的目的.您需要它是反应性的还是静态的?为了节省大量内存，您只需告诉"person.city = city"，它将为每个人事组件注入，但是如果需要进行反应，则需要使用Vue.set -method.请记住，如果每个城市都需要是自己的实例(以便每个人都具有相似的城市对象，但是每个人的属性都需要可编辑)，则需要确保您没有使用引用的对象！因此，您最有可能需要克隆城市对象，这将消耗浏览器的内存.
Also make sure to consider what is the purpose of the sub-object. Do you need it to be reactive, or is it static? In order to save bunch of memory you could just tell "person.city = city", which will be injected for each and every person-component, but if it needs to be reactive then you need to use Vue.set -method... and remember that if each city needs to be own instance (so that each person has similar city-object, but properties need to be editable per person) then you need to make sure that you are not using referred object! Thus you most likely need to clone the city-object, which will eat up browsers memory.
- 您的微组件可能包含分别用于只读状态和编辑器状态的视图状态.这是很常见的.尽管如此，实际上您每次都在创建该微组件的实例，因此数千次初始化该组件.
考虑一下您拥有带有表和表行的类似Excel的电子表格的情况.每个单元格都包含您的自定义"my-input" -component，它从布局中获取"readonly" -property.如果用户界面处于只读状态，则您仅在my-input-component内部显示标签部分，否则，您将显示具有某些特殊条件的输入标签(例如，日期时间，数字，文本，textarea，select-tag等).现在，假设您有100行20列，那么您实际上是在初始化2000 my-input-components.现在的问题是-有什么可以改进的(在性能方面)?
Think of situation where you have Excel-like spreadsheet with table and table-rows. Each cell contains your custom "my-input" -component, which takes "readonly"-property from your layout. If the UI is on the readonly-state then you are displaying only the label part inside that my-input-component, but otherwise you are displaying input-tag with some special conditions (such as having different input for datetime, number, text, textarea, select-tag etc). Now let's assume you have 100 rows with 20 columns, so you are actually initializing 2000 my-input-components. Now the question is -- what could be improved (performance-wise)?
好吧，您可以将readonly-label从my-input-component分离到列表视图，以便显示readonly-version(标签)或显示可编辑的my-input-component.这样，您就具有v-if条件，该条件确保除非您特别要求初始化'em(由于行或整个布局都从readonly-> editable -state移动)，否则不会初始化这2000个微组件.您可能会猜想，当Vue不需要创建2000个组件时，对浏览器的内存影响将是多么大.
Well, you could separate readonly-label from my-input-component to your list-view, so that you either display readonly-version (label) OR you you display the editable my-input-component. This way you have v-if condition, which makes sure that those 2000 micro-components won't be initialized unless you have specifically requested to initialize 'em (due either row or whole layout moving from readonly -> editable -state)... You probably guess how big the impact is memory-wise for browser, when Vue doesn't need to create 2000 components.
如果您的页面加载速度非常慢，则可能根本不是VUE.检查呈现给您的HTML的HTML标签数量.当您有大量标签时，HTML的效果会很差.演示此问题的最简单方法之一是，将具有2000个选项的select-tag重复100次，或者具有一个20000个option-tag.通过具有大量带有不必要包装div等的微组件，您可能会溢出html标签数量.同样，深度和标签越少，浏览器&所需的渲染性能就越低.CPU.
If you are facing that your page loads really slow it might not be VUE at all. Check out the amount of HTML-tags rendered to your HTML. HTML performs rather poorly when you have large amounts of tags. One of the simplest ways to demonstrate this is by repeating select-tag with 2000 options 100 times, or by having a single 20000 option select-tag. The same way you might be overflowing the amount of html-tags by having lots of micro-components with unnecessary wrapping divs etc... The less depth and less tags you have, the less rendering performance is required from browser & CPU.
尝试通过示例学习良好的HTML标签架构.例如，您可以研究Trello -services仪表板视图的编程方式.它是相当半复杂的服务的非常简单漂亮的表示形式，具有最少的细分.
Try to learn good HTML-tag architecture via examples. For an example you could study how Trello -services dashboard-view has been programmed. It's quite simple and beautiful representation of rather semi-complex service, with minimal amount of sub-divs.
有很多方法可以改善内存处理，但是我要说的是，最重要的方法涉及将隐藏"对象与可见对象分离，如我最初的回答所述.第二部分是了解实例对象与引用对象之间的差异.第三是限制对象之间不必要的数据传递.
There are many ways to improve memory handling, but I'd say that most important ones relate to separating "hidden" objects from visible objects, as described on my original answer. Second part is understanding the difference or instanced vs referenced objects. Third is to limit the amount of unnecessary data-passing between objects.
我个人还没有尝试过，但是有一个Vue-virtual-scroller组件可以通过简单地包装看似无限数量的数据来处理任意数量的数据.查看概念@ https://github.com/Akryum/vue-virtual-scroller，让我知道它是否为您解决了问题.
Personally I haven't tried this, but there exists a Vue-virtual-scroller component which handles any amount of data by simply being a wrapper for seemingly infinite amounts of data. Check out the concept @ https://github.com/Akryum/vue-virtual-scroller , and let me know if it solved the problem for you.
我希望这些指南能为您优化组件提供一些想法.永远不要放弃希望，总有改进的空间！
I hope these guidelines give some ideas for optimizing your components. Never give up the hope, there is always room for improvement!
这篇关于如何处理大数据(约50000个对象)的Vue 2内存使用情况的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
