MongoDB oplog 在键名中有带点的记录，无法查询，afaict [英] MongoDB oplog has records with dots in key names, which can't be queried for, afaict

问题描述

鉴于:Mongo 允许使用点"设置嵌套字段，例如:

Given: Mongo allows setting nested fields using "dot"s, e.g.:

rs0:PRIMARY> db.tmp.update({ a: 1 }, { $set: { 'b.c': 2 } }, { upsert: true })
rs0:PRIMARY> db.tmp.findOne()
{
    "_id" : ObjectId("558251c6a3354af70d70f3cc"),
    "a" : 1,
    "b" : {
        "c" : 2
    }
}

在这个例子中，记录是由 upsert 创建的，我可以在 oplog 中验证:

In this example, the record was created by the upsert, which I can verify in the oplog:

rs0:PRIMARY> use local
rs0:PRIMARY> db.oplog.rs.find().sort({ts:-1}).limit(1).pretty()
{
    "ts" : Timestamp(1434603974, 2),
    "h" : NumberLong("2071516013149720999"),
    "v" : 2,
    "op" : "i",
    "ns" : "test.tmp",
    "o" : {
        "_id" : ObjectId("558251c6a3354af70d70f3cc"),
        "a" : 1,
        "b" : {
            "c" : 2
        }
    }
}

当我做同样的事情并且记录只是更新而不是创建时，我似乎得到了相同的行为:

When I do the same thing and the record is merely updated, not created, I seem to get the same behavior:

rs0:PRIMARY> db.tmp.update({ a: 1 }, { $set: { 'b.d': 3 } }, { upsert: true })
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
rs0:PRIMARY> db.tmp.findOne()
{
    "_id" : ObjectId("558251c6a3354af70d70f3cc"),
    "a" : 1,
    "b" : {
        "c" : 2,
        "d" : 3
    }
}

然而，这次 oplog 中的条目结构不同:

However, this time the entry in the oplog is structured differently:

rs0:PRIMARY> use local
rs0:PRIMARY> db.oplog.rs.find().sort({ts:-1}).limit(1).pretty()
{
    "ts" : Timestamp(1434604173, 1),
    "h" : NumberLong("-4353495487634403370"),
    "v" : 2,
    "op" : "u",
    "ns" : "test.tmp",
    "o2" : {
        "_id" : ObjectId("558251c6a3354af70d70f3cc")
    },
    "o" : {
        "$set" : {
            "b.d" : 3
        }
    }
}

(注意 "b.d" 键).

这给我带来了问题，因为我试图通过检查相应的 oplog 条目来调查一些丢失的更新，但 AFAICT 无法查询设置特定嵌套字段的 oplog 条目:

This is causing me issues because I am trying to investigate some dropped updates by inspecting the corresponding oplog entries, but AFAICT there's no way to query for oplog entries that set specific nested fields:

rs0:PRIMARY> db.oplog.rs.findOne({ 'o.$set.b.d': { $exists: true } })
null

是否有任何方法可以查询 oplog 中与特定嵌套字段(在本例中为 b.d)的更新有关的条目?

Is there any way to query the oplog for entries that pertain to updates of a specific nested field (in this case b.d)?

似乎我遇到了 Mongo 的 的不一致应用禁止在字段名称中使用点:一方面我无法创建(通过官方客户端/直接在 Mongo shell 中)或查询它们，但另一方面它是在 oplog 中创建它们，从而无法查询oplog 条目.

It seems that I am running into inconsistent application of Mongo's prohibition of dots in field-names: on one hand I can't create (via official clients / directly in the Mongo shell) or query for them, but on the other it is creating them in the oplog, leaving unqueryable oplog entries.

任何帮助将不胜感激.

为了完整起见，请注意我可以使用包含 $set 位的键成功查询 oplog 条目:

For completeness, note that I can successfully query for oplog entries with keys that include the $set bit:

rs0:PRIMARY> db.tmp.update({ a: 1 }, { $set: { e: 4 } }, { upsert: true })
rs0:PRIMARY> use local
rs0:PRIMARY> db.oplog.rs.findOne({ 'o.$set.e': { $exists: true } })
{
    "ts" : Timestamp(1434604486, 1),
    "h" : NumberLong("1819316318253662899"),
    "v" : 2,
    "op" : "u",
    "ns" : "test.tmp",
    "o2" : {
        "_id" : ObjectId("558251c6a3354af70d70f3cc")
    },
    "o" : {
        "$set" : {
            "e" : 4
        }
    }
}

推荐答案

你说得对，MongoDB 的 oplog 实现中存在一些不一致，它允许每个 op 日志的文档格式在技术上不允许查询这样的文档相应地.

You are correct, there is some inconsistency in MongoDB's oplog implementation that allows a document format for each op log that technically does not allow for such a document to be queried accordingly.

即使插入相同的条目也是不可能的，因为它具有 $set 字段名称:

Even inserting the same entry isn't possible as it has a $set field name:

db.tmp2.insert({ 
    "ts" : Timestamp(1450117240, 1), 
    "h" : NumberLong(2523649590228245285), 
    "v" : NumberInt(2), 
    "op" : "u", 
    "ns" : "test.tmp", 
    "o2" : {
        "_id" : ObjectId("566f069e63d6a355b2c446af")
    }, 
    "o" : {
        "$set" : {
            "b.d" : NumberInt(4)
        }
    }
})

2015-12-14T10:27:04.616-0800 E QUERY    Error: field names cannot start with $ [$set]
    at Error (<anonymous>)
    at DBCollection._validateForStorage (src/mongo/shell/collection.js:161:19)
    at DBCollection._validateForStorage (src/mongo/shell/collection.js:165:18)
    at insert (src/mongo/shell/bulk_api.js:646:20)
    at DBCollection.insert (src/mongo/shell/collection.js:243:18)
    at (shell):1:9 at src/mongo/shell/collection.js:161

并且 b.d 对密钥无效

and the b.d is invalid for a key

db.tmp.update({ a: 1 }, { $set: { 'b.d': 4 } }, { upsert: true })
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

db.oplog.rs.find()


db.tmp2.insert({ 
    "ts" : Timestamp(1450117240, 1), 
    "h" : NumberLong(2523649590228245285), 
    "v" : NumberInt(2), 
    "op" : "u", 
    "ns" : "test.tmp", 
    "o2" : {
        "_id" : ObjectId("566f069e63d6a355b2c446af")
    }, 
    "o" : {
        "set" : {
            "b.d" : NumberInt(4)
        }
    }
})

2015-12-14T10:23:26.491-0800 E QUERY    Error: can't have . in field names [b.d]
    at Error (<anonymous>)
    at DBCollection._validateForStorage (src/mongo/shell/collection.js:157:19)
    at DBCollection._validateForStorage (src/mongo/shell/collection.js:165:18)
    at DBCollection._validateForStorage (src/mongo/shell/collection.js:165:18)
    at insert (src/mongo/shell/bulk_api.js:646:20)
    at DBCollection.insert (src/mongo/shell/collection.js:243:18)
    at (shell):1:9 at src/mongo/shell/collection.js:157

也许应该记录一个 Jira 问题，建议将 $set 搜索的语法设置为值:

Perhaps a Jira issue should be logged that recommends a syntax with the $set search should be set as a value:

{ 
    "ts" : Timestamp(1450117240, 1), 
    "h" : NumberLong(2523649590228245285), 
    "v" : NumberInt(2), 
    "op" : "u", 
    "ns" : "test.tmp", 
    "o2" : {
        "_id" : ObjectId("566f069e63d6a355b2c446af")
    }, 
    "o" : {
        "$set" : {
            "key" : "b.d"
            "value" : NumberInt(4)
        }
    }
}

更新:为此创建了一个 Jira 问题:

Update: Created a Jira issue for this:

https://jira.mongodb.org/browse/SERVER-21889

这篇关于MongoDB oplog 在键名中有带点的记录，无法查询，afaict的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！