在Firebase中查找集合长度 [英] Finding the collection length in firebase

问题描述

我的Firebase实时数据库中有超过2万个对象.我现在需要取出所有这些对象并对它们进行处理.问题是每次我执行服务器时内存不足.这是我当前的代码:

sendEmail.get('/:types/:message', cors(), async (req, res, next) => {
    console.log(5);
    const types = JSON.parse(req.params.types);
    console.log('types', types);
    let recipients = [];
    let mails = [];
    if (types.includes('students')) {
        console.log(1);
        const tmpUsers = await admin.database().ref('Users').orderByChild('student').equalTo(true).once('value').then(r => r.val()).catch(e => console.log(e));
        recipients = recipients.concat(tmpUsers);
    }
    if (types.includes('solvers')) {
        console.log(2);
        let tmpUsers = await admin.database().ref('Users').orderByChild('userType').equalTo('person').once('value').then(r => r.val()).catch(e => console.log(e));
        tmpUsers = tmpUsers.concat(arrayFromObject(await admin.database().ref('Users').orderByChild('userType').equalTo('company').once('value').then(r => r.val()).catch(e => console.log(e))));
        recipients = recipients.concat(tmpUsers);
    }
});

所以我有两个选择.流式传输或使用startAt和endAt限制响应.但是要限制响应，我需要知道我到底有多少个对象.为此，我需要下载整个收藏集...您现在看到了我的问题.如何在不下载整个文档集的情况下了解我有多少文档?

解决方案

您可以尝试通过组合 解决方案

You could try paginating your query by combining limitToFirst/limitToLast and startAt/endAt.

For example, you could perform the first query with limitToFirst(1000), then obtain the last key from this returned list and use that with startAt(key) and another limitToFirst(1000), repeating until you reach the end of the collection.

In node.js, it might look something like this (untested code):

let recipients = [];

let tmpUsers = next();
recipients = filter(recipients, tmpUsers);

// startAt is inclusive, so when this reaches the last result there will only be 1
while (tmpUsers.length>1) {
    let lastKey = tmpUsers.slice(-1).pop().key;
    tmpUsers = next(lastKey);
    if (tmpUsers.length>1) { // Avoid duplicating last result
        recipients = filter(recipients, tmpUsers);
    }
}

async function next(startAt) {
    if (!startAt) {
        return await admin.database().ref('Users')
                .orderByKey()
                .limitToFirst(1000)
                .once('value').then(r => r.val()).catch(e => console.log(e));
    } else {
        return await admin.database().ref('Users')
                .orderByKey()
                .startAt(startAt)
                .limitToFirst(1000)
                .once('value').then(r => r.val()).catch(e => console.log(e));
    }
}

function filter(array1, array2) {
    // TODO: Filter the results here as we can't combine orderByChild/orderByKey
    return array1.concat(array2);
}

The problem with this is that you won't be able to use database-side filtering, so you'd need to filter the results manually, which might make things worse, depending on how many items you need to keep in the recipients variable at a time.

Another option would be to process them in batches (of 1000 for example), pop them from the recipients array to free up resources and then move onto the next batch. It does depend entirely on what actions you need to perform on the objects, and you'll need to weigh up whether it's actually necessary to process (and keep in memory) the entire result set in one go.

这篇关于在Firebase中查找集合长度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！