在Firestore数据库中一次执行500多次操作 [英] Execute more than 500 operations at once in Firestore Database

问题描述

我试图创建一个WriteBatch来控制我的数据库中的一个动态引用.我的应用程序有一个简单的User-Follow-Post-Feed模型，我希望用户在其供稿中查看他关注的所有用户的帖子.在研究Firebase示例(如 Firefeed )和大量关于Stack Overflow的帖子之后，我正在做什么.

I'm trying to create a WriteBatch to keep control of one of my dynamic references in my database. My app have a simple User-Follow-Post-Feed model where I want my user to see in his feed the posts of all the users he is following. What I'm doing after research on Firebase examples (as Firefeed ) and a lot of posts on Stack Overflow.

最佳方法是保留一个路径(在本例中为collection)，在该路径中，我存储用户应在其供稿中看到的帖子的Ids，这意味着应控制复制并删除所有他关注/取消关注的用户.

The optimal idea is keep a path (collection in this case) where I store the Ids of the posts that my user should see in his feed, which means keep control of copy and delete every post of all the users that he follow/unfollow.

我将我的Cloud functions设置为以原子方式保存，并且一切正常，但是当我尝试进行大规模测试时，为用户添加了5000多个帖子，以尝试关注他(请参阅参考资料) Cloud function需要多少时间)，我看到批处理的操作数限制为500.因此，我要做的就是将我的5000个id分成多个小列表，并在每个列表中执行一批，而从未超过500个限制.

I made my Cloud functions for keep this in an atomic way, and everything is working fine, but when I tried to do a massive test, adding more than 5000 posts for an user an trying to follow him (looking for see how much time the Cloud function would take), I saw that batches have a limit of 500 operations. So what I did is split my 5000 id's in multiple small lists and execute one batch per list, never surpassing the 500 limit.

但是即使以这种方式进行操作，我仍然会得到一个错误，提示I can't do more than 500 operations in a single commit，我不知道是否是因为批处理同时执行，或者为什么.我认为也许我可以一个接一个地衔接，并避免一次执行所有命令.但是我仍然有一些麻烦.这就是我提出问题的原因.

But even doing it on this way, I still getting an error that I can't do more than 500 operations in a single commit, I don't know if maybe is because the batches are executing at the same time, or why. I think that maybe I can concat one after another, and avoid execute them all at once. But I still having some troubles with it. So that's the reason of my question.

这是我的方法:

 fun updateFeedAfterUserfollow(postIds: QuerySnapshot, userId: String) {
        //If there is no posts from the followed user, return
        if (postIds.isEmpty) return
        val listOfPostsId = postIds.map { it.id }
        val mapOfInfo = postIds.map { it.id to it.toObject(PublicUserData::class.java) }.toMap()

        //Get User ref
        val ref = firestore.collection(PRIVATE_USER_DATA).document(userId).collection(FEED)
        //Split the list in multiple list to avoid the max 500 operations per batch
        val idsPartition = Lists.partition(listOfPostsId, 400)

        //Create a batch with max 400 operations and execute it until the whole list have been updated
        idsPartition.forEach { Ids ->
            val batch = firestore.batch().also { batch ->
                Ids.forEach { id -> batch.set(ref.document(id), mapOfInfo[id]!!) }
            }
            batch.commit().addOnCompleteListener {
                if (it.isSuccessful)
                    Grove.d { "Commit updated successfully" }
                else Grove.d { "Commit fail" }
            }
        }
    }

推荐答案

最后，问题是由于我试图在事务中实现此批处理操作而产生的，该事务在最后也像批处理一样.这就是为什么即使我为每400个引用生成批处理，这些实例也是在一个事务内创建的，它的计数就像一个超过500个限制的大事务一样.

Finally the problem was caused because I was trying to realize this batch operations inside a transaction, which also acts like a batch in the end. That's why even I was generating batches for each 400 references, these instances were created inside a transaction and it counts like a single big transaction which surpass the 500 limit.

我做了一些更改，并在GitHub上的存储库中实现.

I made a few changes and implemented on a repository on my GitHub.

//Generate the right amount of batches
    const batches = _.chunk(updateReferences, MAX_BATCH_SIZE)
        .map(dataRefs => {
            const writeBatch = firestoreInstance.batch();
            dataRefs.forEach(ref => {
                writeBatch.update(ref, 'author', newAuthor);
            });
            return writeBatch.commit();
        });

它是用打字稿写的，但是您一定会理解的: https://github.com/FrangSierra/firestore-cloud-functions-typescript/blob/master/functions/src/atomic-operations/index.ts

It is writed on typescript, but you will understand it for sure: https://github.com/FrangSierra/firestore-cloud-functions-typescript/blob/master/functions/src/atomic-operations/index.ts

这篇关于在Firestore数据库中一次执行500多次操作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！