随着记录的增长，mongoldb文档更新的性能下降 [英] Degrading performance of mongoldb document updates as record grows

查看：52
本文介绍了随着记录的增长，mongoldb文档更新的性能下降的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我有一个iOS应用程序，它将批数据发送到API端点，该端点将数据存储到mongodb数据库中.我的数据建模如下:

I have an iOS application which sends batches of data to an API endpoint which stores the data into a mongodb database. My data is modeled like:

{
"_id" : ObjectId,
"device_id" : Uuid,
"rtfb_status": bool,
"repetitions" : [
    {
        "session_id" : Uuid,
        "set_id" : Uuid,
        "level" : String,
        "exercise" : String,
        "number" : i32,
        "rom" : f64,
        "duration" : f64,
        "time" : i64
    },
    ...,
],
"imu_data": [
    {
        "session_id": Uuid,
        "data": [
            {
                "acc" : {
                    "y" : f64,
                    "z" : f64,
                    "x" : f64,
                    "time" : i64,
                },
                "gyro" : {
                    "y" : f64,
                    "z" : f64,
                    "x" : f64,
                    "time" : i64,
                }
            },
            ...,
        ]
    },
    ...,
]
}

我的应用程序只是追加到相关的数组上.

My application just appends to the relevant array.

async fn append_to_list<S: Serialize + From<I>, I>(
    self,
    collection: Collection,
    source: I,
    field_name: &str,
) -> Result<i64, CollectionError> {
    let new_records =
        bson::to_bson(&S::from(source)).map_err(CollectionError::DbSerializationError)?;

    self.update_user_record(
        collection,
        bson::doc! { "$push": { field_name: new_records } },
    )
    .await
}

async fn update_user_record(
    self,
    collection: Collection,
    document: bson::Document,
) -> Result<i64, CollectionError> {
    let query = self.try_into()?;

    let update_options = mongodb::options::UpdateOptions::builder()
        .upsert(true)
        .build();

    let updated_res = collection
        .update_one(query, document, update_options)
        .await
        .map_err(CollectionError::DbError)?;

    Ok(updated_res.modified_count)
}

pub async fn add_imu_records(
    self,
    collection: Collection,
    imurecords: JsonImuRecordSet,
) -> Result<i64, CollectionError > {
    self.append_to_list::<ImuDataUpdate, _>(collection, imurecords, "imu_data")
        .await
}

一切正常 ，但是随着时间的流逝，写入性能会下降.从我的应用程序的记录器输出中:

Everything is working, but write performance drops off as time goes on. From the logger output of my application:

 INFO  data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 16.78034ms
 INFO  data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.737755ms
 INFO  data_server > 127.0.0.1:50817 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.143721ms
 INFO  data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 5.021643ms
 INFO  data_server > 127.0.0.1:50818 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.644989ms
 INFO  data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 4.456604ms
 INFO  data_server > 127.0.0.1:50817 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 2.822192ms
 INFO  data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 1.820112ms
 INFO  data_server > 127.0.0.1:50818 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 1.850234ms
 INFO  data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 1.801561ms
 INFO  data_server > 127.0.0.1:50789 "PUT /v1/add_imu_records HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 26.722725ms

注意:add_imu_records调用是一个更大的有效负载，因此我希望它能运行更长的时间.

Note: The add_imu_records call is a much larger payload so I expect it to run longer.

但是在相对较短的时间(可能是10分钟左右)之后，写入操作花费的时间就更长了

But after a relatively short duration (maybe 10 minutes or so) the writes are taking MUCH longer

INFO  data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 23.000502ms
INFO  data_server > 127.0.0.1:50818 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 23.23503ms
INFO  data_server > 127.0.0.1:50789 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 114.679434ms
INFO  data_server > 127.0.0.1:50817 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 143.392153ms
INFO  data_server > 127.0.0.1:50816 "PUT /v1/add_repetition HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 65.101141ms
INFO  data_server > 127.0.0.1:50818 "PUT /v1/add_imu_records HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 117.456596ms

我做错什么了吗?mongodb只是错误的工具，我应该使用RDBMS吗?我有一个分支可以在Postgres上运行，并且响应时间在最佳状态下要比mongo时间慢，但是它们仍然相当稳定.

Am I doing something wrong? Is mongodb just the wrong tool and I should use an RDBMS? I have a branch of this that runs on Postgres, and the response times are slower than the mongo times at their best, but they remain pretty stable.

INFO  data_server > 172.17.0.1:54918 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 7.300945ms
 INFO  data_server > 172.17.0.1:54906 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 5.927394ms
 INFO  data_server > 172.17.0.1:54910 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 6.025674ms
 INFO  data_server > 172.17.0.1:54914 "PUT /v1/add_imu_records HTTP/1.1" 200 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 45.430983ms
 INFO  data_server > 172.17.0.1:54906 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 11.442257ms
 INFO  data_server > 172.17.0.1:54910 "PUT /v1/add_repetition HTTP/1.1" 201 "-" "client/2.0 (edu.odu.cs.nightly; build:385; iOS 13.7.0) Alamofire/5.2.1" 6.875235ms

Mongo在Docker容器中的机器上运行.根据 Object.bsonsize ，该文档为4484480字节(4.48448 Mb).

Mongo is running on my machine in a docker container. According to Object.bsonsize the document is 4484480 bytes (4.48448 Mb).

推荐答案

为了更新文档，MongoDB必须从磁盘中获取整个文档(除非它已经在缓存中)，然后在内存中对其进行修改并写入它回到磁盘.这些修改也将写入操作日志中，以复制到辅助节点.

In order to update a document, MongoDB must fetch the entire document from disk (unless it is already in the cache), then mutate it in memory, and write it back to the disk. The modifications are also written to the oplog for replication to the secondary nodes.

随着文档大小的增加，每个文档将花费更长的时间，并且由于每个文档都占用了越来越大的内存空间，因此高速缓存流失也会开始吞噬无关查询的性能.

As the document size grows, this will take longer for each document, and since each document consumes increasing space in memory, cache churn will also begin to eat away performance for unrelated queries.

MongoDB中的最大文档大小为16MB.如果此文档在10分钟后已经达到4MB，则需要尽快拆分或切断.

The maximum document size in MongoDB is 16MB. If this document is already at 4MB after 10 minutes, it will need to be split or cutoff soon.

这篇关于随着记录的增长，mongoldb文档更新的性能下降的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆