在 Firestore 数据库中一次执行 500 多个操作 [英] Execute more than 500 operations at once in Firestore Database

问题描述

我正在尝试创建一个 WriteBatch 来控制我的数据库中的动态引用之一.我的应用程序有一个简单的 User-Follow-Post-Feed 模型，我希望我的用户在他的提要中看到他关注的所有用户的帖子.在研究 Firebase 示例(如 Firefeed )和 Stack Overflow 上的大量帖子之后，我正在做什么.

I'm trying to create a WriteBatch to keep control of one of my dynamic references in my database. My app have a simple User-Follow-Post-Feed model where I want my user to see in his feed the posts of all the users he is following. What I'm doing after research on Firebase examples (as Firefeed ) and a lot of posts on Stack Overflow.

最佳的想法是保留一条路径(在这种情况下为 collection)，我存储用户应该在他的提要中看到的帖子的 Ids，这意味着保留控制复制和删除他关注/取消关注的所有用户的每个帖子.

The optimal idea is keep a path (collection in this case) where I store the Ids of the posts that my user should see in his feed, which means keep control of copy and delete every post of all the users that he follow/unfollow.

我创建了我的 Cloud 函数 以保持这种原子方式，一切正常，但是当我尝试进行大规模测试时，为用户添加了 5000 多个帖子跟着他(看看 Cloud 函数 需要多少时间)，我看到批次有 500 次操作的限制.所以我所做的就是将我的 5000 个 ID 拆分为多个小列表，并在每个列表中执行一批，永远不会超过 500 个限制.

I made my Cloud functions for keep this in an atomic way, and everything is working fine, but when I tried to do a massive test, adding more than 5000 posts for an user an trying to follow him (looking for see how much time the Cloud function would take), I saw that batches have a limit of 500 operations. So what I did is split my 5000 id's in multiple small lists and execute one batch per list, never surpassing the 500 limit.

但即使这样做，我仍然得到一个错误 我不能在一次提交中执行超过 500 次操作，我不知道是否可能是因为批次是同时执行，或者为什么.我认为也许我可以一个接一个地连接，并避免一次执行它们.但我仍然有一些麻烦.所以这就是我提出问题的原因.

But even doing it on this way, I still getting an error that I can't do more than 500 operations in a single commit, I don't know if maybe is because the batches are executing at the same time, or why. I think that maybe I can concat one after another, and avoid execute them all at once. But I still having some troubles with it. So that's the reason of my question.

这是我的方法:

 fun updateFeedAfterUserfollow(postIds: QuerySnapshot, userId: String) {
        //If there is no posts from the followed user, return
        if (postIds.isEmpty) return
        val listOfPostsId = postIds.map { it.id }
        val mapOfInfo = postIds.map { it.id to it.toObject(PublicUserData::class.java) }.toMap()

        //Get User ref
        val ref = firestore.collection(PRIVATE_USER_DATA).document(userId).collection(FEED)
        //Split the list in multiple list to avoid the max 500 operations per batch
        val idsPartition = Lists.partition(listOfPostsId, 400)

        //Create a batch with max 400 operations and execute it until the whole list have been updated
        idsPartition.forEach { Ids ->
            val batch = firestore.batch().also { batch ->
                Ids.forEach { id -> batch.set(ref.document(id), mapOfInfo[id]!!) }
            }
            batch.commit().addOnCompleteListener {
                if (it.isSuccessful)
                    Grove.d { "Commit updated successfully" }
                else Grove.d { "Commit fail" }
            }
        }
    }

推荐答案

最后问题是因为我试图在一个事务中实现这个批处理操作，最终也像批处理一样.这就是为什么即使我为每 400 个引用生成批次，这些实例是在一个事务中创建的，它就像一个超过 500 个限制的单个大事务.

Finally the problem was caused because I was trying to realize this batch operations inside a transaction, which also acts like a batch in the end. That's why even I was generating batches for each 400 references, these instances were created inside a transaction and it counts like a single big transaction which surpass the 500 limit.

我做了一些更改并在我的 GitHub 上的存储库中实现.

I made a few changes and implemented on a repository on my GitHub.

//Generate the right amount of batches
    const batches = _.chunk(updateReferences, MAX_BATCH_SIZE)
        .map(dataRefs => {
            const writeBatch = firestoreInstance.batch();
            dataRefs.forEach(ref => {
                writeBatch.update(ref, 'author', newAuthor);
            });
            return writeBatch.commit();
        });

它是在打字稿上写的，但你肯定会理解它:https://github.com/FrangSierra/firestore-cloud-functions-typescript/blob/master/functions/src/atomic-operations/index.ts

It is writed on typescript, but you will understand it for sure: https://github.com/FrangSierra/firestore-cloud-functions-typescript/blob/master/functions/src/atomic-operations/index.ts

这篇关于在 Firestore 数据库中一次执行 500 多个操作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！