在 Firestore 数据库中一次执行 500 多项操作 [英] Execute more than 500 operations at once in Firestore Database

问题描述

我正在尝试创建一个 WriteBatch 来控制我的数据库中的一个动态引用.我的应用程序有一个简单的 User-Follow-Post-Feed 模型，我希望我的用户在他的订阅源中看到他关注的所有用户的帖子.在研究了 Firebase 示例(如 Firefeed )和 Stack Overflow 上的大量帖子后，我正在做什么.

I'm trying to create a WriteBatch to keep control of one of my dynamic references in my database. My app have a simple User-Follow-Post-Feed model where I want my user to see in his feed the posts of all the users he is following. What I'm doing after research on Firebase examples (as Firefeed ) and a lot of posts on Stack Overflow.

最佳想法是保留一条路径(在本例中为 collection)，我将在其中存储用户应该在其提要中看到的帖子的 Ids，这意味着保持控制复制和删除他关注/取消关注的所有用户的每个帖子.

The optimal idea is keep a path (collection in this case) where I store the Ids of the posts that my user should see in his feed, which means keep control of copy and delete every post of all the users that he follow/unfollow.

我制作了我的 Cloud 函数 以保持原子性，并且一切正常，但是当我尝试进行大规模测试时，为用户添加了 5000 多个帖子并试图跟着他(看看Cloud function 需要多少时间)，我看到批处理有 500 次操作的限制.所以我所做的是将我的 5000 个 id 分成多个小列表，并为每个列表执行一批，永远不会超过 500 的限制.

I made my Cloud functions for keep this in an atomic way, and everything is working fine, but when I tried to do a massive test, adding more than 5000 posts for an user an trying to follow him (looking for see how much time the Cloud function would take), I saw that batches have a limit of 500 operations. So what I did is split my 5000 id's in multiple small lists and execute one batch per list, never surpassing the 500 limit.

但即使这样做，我仍然得到一个错误，我不能在一次提交中做超过 500 个操作，我不知道是否可能是因为批次是同时执行，或者为什么.我想也许我可以一个接一个地连接，避免一次执行它们.但我仍然有一些麻烦.所以这就是我提出问题的原因.

But even doing it on this way, I still getting an error that I can't do more than 500 operations in a single commit, I don't know if maybe is because the batches are executing at the same time, or why. I think that maybe I can concat one after another, and avoid execute them all at once. But I still having some troubles with it. So that's the reason of my question.

这是我的方法:

 fun updateFeedAfterUserfollow(postIds: QuerySnapshot, userId: String) {
        //If there is no posts from the followed user, return
        if (postIds.isEmpty) return
        val listOfPostsId = postIds.map { it.id }
        val mapOfInfo = postIds.map { it.id to it.toObject(PublicUserData::class.java) }.toMap()

        //Get User ref
        val ref = firestore.collection(PRIVATE_USER_DATA).document(userId).collection(FEED)
        //Split the list in multiple list to avoid the max 500 operations per batch
        val idsPartition = Lists.partition(listOfPostsId, 400)

        //Create a batch with max 400 operations and execute it until the whole list have been updated
        idsPartition.forEach { Ids ->
            val batch = firestore.batch().also { batch ->
                Ids.forEach { id -> batch.set(ref.document(id), mapOfInfo[id]!!) }
            }
            batch.commit().addOnCompleteListener {
                if (it.isSuccessful)
                    Grove.d { "Commit updated successfully" }
                else Grove.d { "Commit fail" }
            }
        }
    }

推荐答案

最后问题是因为我试图在一个事务中实现这个批处理操作，它最终也像一个批处理.这就是为什么即使我也为每 400 个引用生成批处理，这些实例是在事务中创建的，它就像一个超过 500 限制的单个大事务.

Finally the problem was caused because I was trying to realize this batch operations inside a transaction, which also acts like a batch in the end. That's why even I was generating batches for each 400 references, these instances were created inside a transaction and it counts like a single big transaction which surpass the 500 limit.

我做了一些更改并在我的 GitHub 上的存储库中实施.

I made a few changes and implemented on a repository on my GitHub.

//Generate the right amount of batches
    const batches = _.chunk(updateReferences, MAX_BATCH_SIZE)
        .map(dataRefs => {
            const writeBatch = firestoreInstance.batch();
            dataRefs.forEach(ref => {
                writeBatch.update(ref, 'author', newAuthor);
            });
            return writeBatch.commit();
        });

它是写在打字稿上的，但你肯定会理解它:https://github.com/FrangSierra/firestore-cloud-functions-typescript/blob/master/functions/src/atomic-operations/index.ts

It is writed on typescript, but you will understand it for sure: https://github.com/FrangSierra/firestore-cloud-functions-typescript/blob/master/functions/src/atomic-operations/index.ts

这篇关于在 Firestore 数据库中一次执行 500 多项操作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！