CouchDB 2中的序列号错误，还是还有另一种比较序列号的方法? [英] Sequence number bug in CouchDB 2 or is there another way to compare sequence numbers?

问题描述

我正在更深入地研究CouchDB 2，并且发现一些意外的序列号排序.在一种情况下，我发现_changes提要中的早期更改具有序列号

<预> <代码> 99-g1AAAAI-eJyd0EsOgjAQBuAGiI-dN9C9LmrBwqzkJtrSNkgQV6z1JnoTvYneBEvbhA0aMU1mkj6-_NMSITTJfYFm2anOcsFT10mpTzyG-LxpmiL32eqoN8aEAcWE9dz_jPCFrnzrHGQchiFM4kSgaV0JqQ6VFF-AtAV2DggMgCEGxrNhQfatc3bOyDiKUalg2EBVoCu66KapazcUh41e69-GssjNIvcWWRokk2oNofwj0MNazy4QFURhGQ0J9LKI-SHPIBHEgiak51nxBhxnrRk

对于同一数据库，我_changes提要中的最后一个序列号是

<预> <代码> 228-g1AAAAJFeJyd0EkOgjAUBuAGTJCdN9AjlIKFruQm2jFAEFes9SZ6E72J3gQ7JW7QCGnyXtLhy-vfAgCWVSjAip96XglW-o5afRJQwNbDMDRVSOuj3ogQJRgiOnL_O8I2urKdd4B1KCRpkRcCxH0npKo7KX4ApQH2HogsAElOKOPTBjkY5-yd2DqKYqnItA91C13BRTdNXY0VWouRrV7JDOvmrLuxlLW4VAlJ5Qzr4aznJ2wskIIy-y9sh7wcYoMKLJKRXOACjTxr3uHcsBE

在浏览器控制台中，以下内容为假

<预> <代码> '228-g1AAAAJFeJyd0EkOgjAUBuAGTJCdN9AjlIKFruQm2jFAEFes9SZ6E72J3gQ7JW7QCGnyXtLhy-vfAgCWVSjAip96XglW-o5afRJQwNbDMDRVSOuj3ogQJRgiOnL_O8I2urKdd4B1KCRpkRcCxH0npKo7KX4ApQH2HogsAElOKOPTBjkY5-yd2DqKYqnItA91C13BRTdNXY0VWouRrV7JDOvmrLuxlLW4VAlJ5Qzr4aznJ2wskIIy-y9sh7wcYoMKLJKRXOACjTxr3uHcsBE' >'99 -g1AAAAI-eJyd0EsOgjAQBuAGiI-dN9C9LmrBwqzkJtrSNkgQV6z1JnoTvYneBEvbhA0aMU1mkj6-_NMSITTJfYFm2anOcsFT10mpTzyG-LxpmiL32eqoN8aEAcWE9dz_jPCFrnzrHGQchiFM4kSgaV0JqQ6VFF-AtAV2DggMgCEGxrNhQfatc3bOyDiKUalg2EBVoCu66KapazcUh41e69-GssjNIvcWWRokk2oNofwj0MNazy4QFURhGQ0J9LKI-SHPIBHEgiak51nxBhxnrRk"

这是一个错误还是我需要使用其他方法来比较序列号?

在查看我的_changes提要中的其他序列号时，看起来它们通常按照我期望的顺序排序，但是在这种情况下，当第一个数字(例如99，从2位跳到3位，顺序中断.如果将其简化为一个简单的字符串比较示例，您会看到'228'>'99'=> false

解决方案

以下答案包含带有@rnewson的电子邮件线程的摘录.我希望它可以帮助其他人了解CouchDB 2中的序列号.谢谢罗伯特！

背景:

没有简单的方法可以在2.0中对其进行比较，也没有要求保持秩序.简而言之，它们并非旨在进行检查或在couchdb之外进行比较；不透明地对待他们.

前面的数字是各个更新序列的总和在第二部分中进行了编码，并且仅用于欺骗较旧的版本长沙发复制器进入检查点.

序列字符串的后半部分是{node，范围，seq}元组(其中seq是您从中知道的整数值2.0之前的版本).当序列字符串传回时，作为since =参数，ouchdb对该字符串进行解码并传递适用于各个分片的整数seq值.

总而言之，前台人数应该增加.完整的字符串本身不可比较，因为没有定义的顺序到编码列表(因此可以生成两个字符串，分别是编码不同，但解码为相同的元组列表，只是在不同的顺序).

另一方面，更改供稿并不完全下令.对于给定的分片，它是 完全有序的(一个分片是与具有整数序列的2.0版之前的数据库完全相同)，couchdb不会混洗该输出(尽管复制的正确性会如果确实如此，则将其保留).集群数据库由多个数据库组成分片("q"值，默认为4 iirc).集群的更改供稿将这些单独的更改供稿合并为一个，但不会为此施加总订单.我们不做因为它会很昂贵且不必要.

该解决方案，如果您需要侦听_changes提要，然后重新启动从您以后停下来的地方开始:

正确使用更改供稿的算法是:

1. 读取/dbname/_changes
2. 幂等地处理每一行
3. 定期(每X秒或每X行)存储您处理的最后一行的"seq"值

如果您崩溃了，或者您没有使用continuous = true，则可以执行再次执行相同的步骤，但在步骤1中进行了修改；

已修订1.阅读/dbname/_changes?since = X

其中X是您在第3步中保存的值.连续模式，那么您只需将"last_seq"值记录在结束非连续响应的结束.您冒着以下风险不过，还会处理更多的项目.

使用此方案(复制器和所有索引器都遵循)，您可以不在乎结果是否有问题，您不需要比较任何两个seq值.

您要做需要确保您可以正确处理相同的更改多次.举一个例子，考虑复制器，当它从更改提要中看到一行，并询问目标数据库是否包含该行中的_id和_rev值.如果是这样，复制器移至下一行.如果没有，它将尝试写该行中的文档到目标数据库.如果发生崩溃，因此使用之前的seq值调用_changes处理该行时，它将询问目标数据库是否具有_id/_rev再一次，只有这次目标才会说是.

I’m digging deeper into CouchDB 2 and I’m finding some unexpected ordering with sequence numbers. In one case, I found that an early change in a _changes feed has the sequence number

99-g1AAAAI-eJyd0EsOgjAQBuAGiI-dN9C9LmrBwqzkJtrSNkgQV6z1JnoTvYneBEvbhA0aMU1mkj6-_NMSITTJfYFm2anOcsFT10mpTzyG-LxpmiL32eqoN8aEAcWE9dz_jPCFrnzrHGQchiFM4kSgaV0JqQ6VFF-AtAV2DggMgCEGxrNhQfatc3bOyDiKUalg2EBVoCu66KapazcUh41e69-GssjNIvcWWRokk2oNofwj0MNazy4QFURhGQ0J9LKI-SHPIBHEgiak51nxBhxnrRk

The last sequence number in my _changes feed, for the same DB, is

228-g1AAAAJFeJyd0EkOgjAUBuAGTJCdN9AjlIKFruQm2jFAEFes9SZ6E72J3gQ7JW7QCGnyXtLhy-vfAgCWVSjAip96XglW-o5afRJQwNbDMDRVSOuj3ogQJRgiOnL_O8I2urKdd4B1KCRpkRcCxH0npKo7KX4ApQH2HogsAElOKOPTBjkY5-yd2DqKYqnItA91C13BRTdNXY0VWouRrV7JDOvmrLuxlLW4VAlJ5Qzr4aznJ2wskIIy-y9sh7wcYoMKLJKRXOACjTxr3uHcsBE

In a browser console, the following is false

'228-g1AAAAJFeJyd0EkOgjAUBuAGTJCdN9AjlIKFruQm2jFAEFes9SZ6E72J3gQ7JW7QCGnyXtLhy-vfAgCWVSjAip96XglW-o5afRJQwNbDMDRVSOuj3ogQJRgiOnL_O8I2urKdd4B1KCRpkRcCxH0npKo7KX4ApQH2HogsAElOKOPTBjkY5-yd2DqKYqnItA91C13BRTdNXY0VWouRrV7JDOvmrLuxlLW4VAlJ5Qzr4aznJ2wskIIy-y9sh7wcYoMKLJKRXOACjTxr3uHcsBE' > '99-g1AAAAI-eJyd0EsOgjAQBuAGiI-dN9C9LmrBwqzkJtrSNkgQV6z1JnoTvYneBEvbhA0aMU1mkj6-_NMSITTJfYFm2anOcsFT10mpTzyG-LxpmiL32eqoN8aEAcWE9dz_jPCFrnzrHGQchiFM4kSgaV0JqQ6VFF-AtAV2DggMgCEGxrNhQfatc3bOyDiKUalg2EBVoCu66KapazcUh41e69-GssjNIvcWWRokk2oNofwj0MNazy4QFURhGQ0J9LKI-SHPIBHEgiak51nxBhxnrRk'

Is this a bug or do I need to use some other method to compare sequence numbers?

In looking at the other sequence numbers in my _changes feed, it looks like they are generally ordered as I would expect, but in this case it appears that when the first number, e.g. 99, jumps from 2 digits to 3 digits, the ordering breaks. If you boil this down to a simple string comparison example, you can see that '228' > '99' => false

解决方案

The following answer contains excerpts from an email thread with @rnewson. I hope it helps someone else to understand sequence numbers in CouchDB 2. Thanks, Robert!

The background:

There's no easy way to compare them in 2.0 and no requirement for them to be in order. They are not, in short, designed to be examined or compared outside of couchdb; treat them opaquely.

The number on the front is the sum of the individual update sequences encoded in the second part and exists only to trick older versions of the couchdb replicator into making checkpoints.

The latter half of the sequence string is an encoded list of {node, range, seq} tuples (where seq is the integer value you know from pre-2.0 releases). When a sequence string is passed back in, as the since= parameter, couchdb decodes this string and passes the appropriate integer seq value to the individual shard.

All that said, in general the front number should increase. The full strings themselves are not comparable, since there's no defined order to the encoded list (so two strings could be generated that are encoded differently but decode to the same list of tuples, just in a different order).

Another aspect to this is that the changes feed is not totally ordered. For a given shard it is totally ordered (a shard being identical to a pre 2.0 database with an integer sequence), couchdb doesn't shuffle that output (though correctness of replication would be retained if it did). A clustered database is comprised of several shards, though (the 'q' value, defaulting to 4 iirc). The clustered changes feed combines those separate changes feed into a single one, but makes no effort to impose a total order over that. We don't do it because it would be expensive and unnecessary.

The solution if you need to listen on a _changes feed and then restart from where you left off later:

The algorithm for correctly consuming the changes feed is:

1. read /dbname/_changes
2. process each row idempotently
3. periodically (every X seconds or every X rows) store the "seq" value of the last row you processed

If you ever crash, or if you weren't using continuous=true, you can do this same procedure again but modified in step 1;

revised 1. read /dbname/_changes?since=X

where X is the value you saved in step 3. If you're not using continuous mode then you could just record the "last_seq" value at the end of consuming the non-continuous response. You run the risk of reprocessing a lot more items, though.

With this scheme (which the replicator and all indexers follow), you don't care if the results come out of order, you don't need to compare any two seq values.

You do need to ensure you can correctly process the same change multiple times. For an example of that, consider the replicator, when it sees a row from a changes feed it asks the target database if it contains the _id and _rev values from that row. If it does, the replicator moves on to the next row. If it doesn't, it tries to write the document in that row to the target database. In the event of a crash, and therefore a call to _changes with a seq value from before processing that row, it will ask the target database if it has the _id/_rev again, only this time the target will say yes.

这篇关于CouchDB 2中的序列号错误，还是还有另一种比较序列号的方法?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！