优化的批量（块）上传对象到IndexedDB [英] Optimized Bulk (Chunk) Upload Of Objects Into IndexedDB

问题描述

我想在一个事务中将对象添加到IndexedDB中的某个表中：

I want to add objects into some table in IndexedDB in one transaction:

_that.bulkSet = function(data, key) {
    var transaction = _db.transaction([_tblName], "readwrite"),
        store = transaction.objectStore(_tblName),
        ii = 0;

    _bulkKWVals.push(data);
    _bulkKWKeys.push(key);

    if (_bulkKWVals.length == 3000) {
        insertNext();
    }

    function insertNext() {
        if (ii < _bulkKWVals.length) {
            store.add(_bulkKWVals[ii], _bulkKWKeys[ii]).onsuccess = insertNext;
            ++ii;
        } else {
            console.log(_bulkKWVals.length);
        }
    }
};

看起来它工作正常，但它不是非常优化的方式，特别是如果数字对象非常高（~50.000-500.000）。我怎么可能优化它？理想情况下，我想先添加3000，然后从数组中删除它，然后添加另一个3000，即以块为单位。任何想法？

Looks like that it works fine, but it is not very optimized way of doing that especially if the number of objects is very high (~50.000-500.000). How could I possibly optimize it? Ideally I want to add first 3000, then remove it from the array, then add another 3000, namely in chunks. Any ideas?

推荐答案

连续插入那么多行是不可能获得良好的表现。

Inserting that many rows consecutively, is not possible to get good performance.

我是一个IndexedDB 开发者，并且在您谈论的规模上拥有IndexedDB的实际经验关于（连续写入数十万行）。它不太漂亮。

I'm an IndexedDB dev and have real-world experience with IndexedDB at the scale you're talking about (writing hundreds of thousands of rows consecutively). It ain't too pretty.

在我看来，当必须连续写入大量数据时，IDB不适合使用。如果我要构建一个需要大量数据的IndexedDB应用程序，我会想出一种随着时间的推移慢慢播种的方法。

In my opinion, IDB is not suitable for use when a large amount of data has to be written consecutively. If I were to architect an IndexedDB app that needed lots of data, I would figure out a way to seed it slowly over time.

问题在于写入，我认为问题在于写入的缓慢以及它们的i / o密集性使得随着时间的推移会变得更糟。 （读取总是在IDB快速减轻，因为它的价值。）

The issue is writes, and the problem as I see it is that the slowness of writes, combined with their i/o intensive nature, makes gets worse over time. (Reads are always lightening fast in IDB, for what it's worth.)

首先，你可以通过重新使用交易来节省开支。因此，你的第一直觉可能是尝试将所有东西塞进同一个交易中。但是，例如，我在Chrome中发现的是，浏览器似乎不喜欢长时间运行的写入，可能是因为某些机制意味着限制行为不当的标签。

To start, you'll get savings from re-using transactions. Because of that your first instinct might be to try to cram everything into the same transaction. But from what I've found in Chrome, for example, is that the browser doesn't seem to like long-running writes, perhaps because of some mechanism meant to throttle misbehaving tabs.

我不确定你会看到什么样的表现，但平均数字可能会欺骗你，具体取决于你的测试大小。限制更快的是吞吐量，但如果你试图连续插入大量数据，请特别注意随着时间的推移写入。

I'm not sure what kind of performance you're seeing, but average numbers might fool you depending on the size of your test. The limiting faster is throughput, but if you're trying to insert large amounts of data consecutively pay attention to writes over time specifically.

我碰巧正在进行演示有数十万行可供我使用，并有统计数据。在禁用我的可视化功能的情况下，在IDB上运行纯粹的破折号，这就是我在Chrome 32中在单个对象商店中看到的内容，其中包含一个带有自动递增主键的非唯一索引。

I happen to be working on a demo with several hundred thousand rows at my disposal, and have stats. With my visualization disabled, running pure dash on IDB, here's what I see right now in Chrome 32 on a single object store with a single non-unique index with an auto-incrementing primary key.

一个更小，更小的27k行数据集，我看到60-70个条目/秒：

* ~30秒：平均921个条目/秒（总是有很多插入开始的时候，62 /秒我正在采样

* ~60秒：389 /秒平均值（持续减少开始超过影响初始爆发）71 /秒瞬间

* ~1：30：258 /秒，67 /秒：
* ~2：00（约1/3完成）：平均188 /秒，时刻66 /秒

A much, much smaller 27k row dataset, I saw 60-70 entries/second:
* ~30 seconds: 921 entries/second on average (there's always a great burst of inserts at the start), 62/second at the moment I sampled
* ~60 seconds: 389/second average (sustained decreases starting to outweigh effect initial burst) 71/second at moment
* ~1:30: 258/second, 67/second at moment
* ~2:00 (~1/3 done): 188/second on average, 66/second at moment

具有小得多的数据集的一些示例显示出更好的性能，但具有相似的特征。同样大得多的数据集 - 效果被夸大了，而且我在离开多个小时时看到的每秒只有1个条目。

Some examples with a much smaller dataset show far better performance, but similar characteristics. Ditto much larger datasets - the effects are greatly exaggerated and I've seen as little as <1 entries per second when leaving for multiple hours.

这篇关于优化的批量（块）上传对象到IndexedDB的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！