将项目推入数组的最佳性能方式? [英] The best performant way to push items into array?

问题描述

在我的网站上，我有很多数据数组. 例如:顶点数组，颜色数组，大小数组...

In my website i have many arrays with data. for example: vertices array, colors array, sizes array...

我正在处理大量物品.高达数千万.

I'm working with big amounts of items. Up to tens of millions.

在将数据添加到数组中之前，我需要对其进行处理. 到目前为止，我是在主线程中完成此操作的，这使我的网站冻结了X秒. 由于处理以及将处理后的数据添加到数组中而使其冻结.

Before adding the data into the arrays I need to process it. Until now, I did it in the main thread and this made my website freeze for X seconds. It froze because of the processing and because of adding the processed data into the arrays.

今天，我将处理转移"(做了很多工作)到Web Worker中，但是处理后的数据正在主线程中添加.我设法节省了处理的冻结时间，但没有节省添加的时间.

Today I 'moved' (did a lot of work) the processing into web workers, but the processed data is being added in the main thread. I managed to save the freezing time of the processing but not of the adding.

添加仅通过array.push()或array.splice()完成.

我已经阅读了一些有关数组如何工作的文章，并发现当我们向数组中添加项目时，该数组已被完全复制到内存中具有array.length + 1大小的新位置并在那里添加了值.这使我的数据发送速度变慢.

I've read some articles about how array works, and found out when we add item to an array, the array is being fully copied to a new place in the memory with array.length + 1 size and there adding the value. This makes my data pushing slow.

我还读到类型化数组要快得多.但是为此，我需要知道我不知道的数组大小，并且要创建一个带有额外计数器的大类型数组并管理在中间(而不是数组末尾)添加项目，很多代码更改，我目前不希望这样做.

I also read that typed array are much faster. But for this I would need to know the size of the array, which I don't know, and for creating a big typed array with extra counter and managing adding items in the middle(and not the end of the array) would be a lot of code change, which i don't want to do at this time.

所以，对于我的问题， 我有从网络工作者返回的TypedArray，这需要放入常规数组中.最佳的执行方式是什么? (今天我正在循环运行，一个又一个地推送)

So, for my question, I have TypedArray that return from the web worker, and this i need to put into regular array. What is the best performant way to do it? (today i'm running in a loop and pushing one after the other)

编辑

示例网站的工作方式: 客户添加项目数，假设为100000. 项目原始数据将被收集并发送给工作人员. 工作人员正在处理所有信息，并将处理后的数据作为类型数组发送回(用作可传输对象).在主线程中，我们将处理后的数据添加到数组中-在末尾或在某些特定索引中. 第二轮.客户再添加100000个项目.发送给工作程序并将结果添加到主线程数组中. 第三轮可以是10个项目，第四轮可以是10000，第五轮可以删除10-2000的索引，...

Example how the website work: the client add count of items, lets say 100000. The items raw data is being collected and send to the worker. The worker is processing all the information and sending back the processed data as typed-array (for using as transferable objects). In the main thread we are adding the processed data to arrays - to the end or in some specific index. 2nd round. the client add another 100000 items. sending to the worker and the result being added to the main thread arrays. 3nd round can be 10 items, 4nd round 10000, 5nd round can remove indices 10-2000, ...

推荐答案

使用这些评论进行了更多研究，并思考了另一个方向.

Did some more research using the comments and thought about another direction.

我尝试使用typedArray.set方法，发现它非常快.

I've tried using typedArray.set method and discovered that it is very very fast.

1000万个项目花费0.004秒，而array.push 0.866秒.我将1000万个数组分为10个数组，以确保从索引0开始时set方法不能更快地工作.

10 million of items using sets took 0.004 seconds, compares to array.push 0.866 seconds. I separated the the 10 millions into 10 arrays just to make sure the set method is not working faster when starting from index 0.

这样，我想我什至可以使用TypedArray实现自己的insertAtIndex，它将所有项目向前推进并在正确的索引中设置新项目.

This way I think I would even implement my insertAtIndex of my own using the TypedArray, which pushing all the items forward and setting the new one\s in the right index.

此外，我可以使用TypedArray.subArray根据数组中的实际数据量(不复制数据)获取子数据-对于将数据上传到缓冲区(WebGL)很有用

In addition, I can use TypedArray.subArray to fetch my sub data according to the real amount of data in the array (which is not copying the data) - useful for uploading the data to the buffer (WebGL)

我说过我想使用常规数组，但是这种性能提升我认为我不会再明智.当我将MyNewTypedArray包装为具有所有push，splice自己的实现的TypedArray时，并不需要那么多的工作.

I said I want to work with regular arrays but this performance boost I don't think I would get other wise. And it is not so much work, when I'm wrapping MyNewTypedArray as TypedArray with all the push, splice, own implementation.

希望此信息对任何人都有帮助

Hope this info helped anyone

var maxCount = 10000000;
var a = new Float32Array(maxCount);
var aSimple = [];

var arrays = [];
var div = 10;
var arrayLen = maxCount / div;
for (var arraysIdx = 0; arraysIdx < div; arraysIdx++) {
  var b = new Float32Array(arrayLen);
  for (var i = 0; i < b.length; i++) {
    b[i] = i * (arraysIdx + 1);
  }
  arrays.push(b);
}

var timeBefore = new Date().getTime();

for (var currArrayIdx = 0; currArrayIdx < arrays.length; currArrayIdx++) {
  a.set(arrays[currArrayIdx], currArrayIdx * arrayLen);
}
var timeAfter = new Date().getTime();
good.innerHTML = (timeAfter - timeBefore) / 1000 + " sec.\n";
timeBefore = new Date().getTime();
for (var currArrayIdx = 0; currArrayIdx < arrays.length; currArrayIdx++) {
  for (var i = 0; i < arrayLen; i++) {
    aSimple.push(arrays[currArrayIdx][i]);
  }
}

timeAfter = new Date().getTime();
bad.innerHTML = (timeAfter - timeBefore) / 1000 + " sec.\n";

Using set of TypedArray:
<div id='good' style='background-color:lightGreen'>working...</div>
Using push of Array:
<div id='bad' style='background-color:red'>working...</div>

这篇关于将项目推入数组的最佳性能方式?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！