分层压缩策略如何确保90％的读取来自一个稳定状态 [英] How does the Leveled Compaction Strategy ensure 90% of reads are from one sstable

问题描述

我试图了解Cassandra中的 Leveled Compacting Strategy 如何工作，以确保单个sstable可以满足所有读取的90％。

从DataStax Doc：

将新的稳定表添加到第一层L0，并立即用L1中的稳定表进行压缩。当L1填满时，额外的稳定级将提升为L2。 L1中生成的后续sstable将与L2中与其重叠的sstable压缩。

解决方案

LeveledCompactionStrategy（LCS）实现了LevelDB的内部。您可以在 LevelDB实施文档中查看确切的实施详细信息。 / p>

为了给您一个简单的解释，请考虑以下几点：

1. 固定时会创建每个sstable （相对较小）达到大小限制。默认情况下，L0获取5MB的文件文件，并且每个后续级别是文件大小的10倍。 （在L1中，您将拥有50MB的数据，在L2中将具有500MB的数据，依此类推。）


2. 稳定表是通过保证它们不会重叠


3. 当某个级别填满时，将触发压缩，并将稳定级从L级提升到L + 1级。因此，在L1中，约10个文件中有50MB，L2中约100个文件中有500MB，依此类推。

黑体字相关的详细信息是证明90％从同一文件读取（稳定）。让我们一起做数学，一切都会变得更加清晰（我希望：）

想象一下，您在L0中拥有键A，B，C，D，E，每个键占用1MB的数据。

接下来，我们插入键F。由于填充了0级，压缩将创建一个文件，该文件的级别为[A，B，C，D，E]，而F将保持在级别0。

这是L1中1个文件中约83％的数据

接下来，我们插入G，H，I ，J和K。所以L0再次填满，L1用[I，G，H，I，J]得到新的稳定值。

现在我们在L0中有K，[A L1中的、、、 B，C，D，E]和[I，G，H，I，J]

而L1中约有90％的数据 ：）

如果我们继续插入键，我们将得到相同的行为，这就是为什么您从大致相同的文件/稳定表中获得90％的读取结果的原因。

在本段中，我提到的链接提供了更深入，更详细的信息（更新和逻辑删除会发生什么）（压缩选举的大小不同，因为它们是LevelDB的默认值，而不是C * s）：

当级别L的大小超过其限制时，我们将其压缩到后台线程中。压缩从级别L中选择一个文件，并从下一级别L + 1中选择所有重叠的文件。请注意，如果级别L文件仅与级别（L + 1）文件的一部分重叠，则级别（L + 1）的整个文件将用作压缩的输入，并且在压缩后将被丢弃。撇开：因为0级是特殊的（文件中的文件可能会相互重叠），所以我们特别处理从0级到1级的压缩：0级压缩可能会选择多个0级文件，以防其中某些文件文件相互重叠。

压缩会合并选取的文件的内容，以生成一系列级别（L + 1）的文件。当前输出文件达到目标文件大小（2MB）之后，我们切换到生成新的level-（L + 1）文件。当当前输出文件的键范围增长到足以重叠超过十个级别（L + 2）文件时，我们还将切换到新的输出文件。这最后一条规则确保了以后对级别（L + 1）文件的压缩不会从级别（（L + 2））中获取太多数据。

希望这会有所帮助！

I am trying to understand how the Leveled Compaction Strategy in Cassandra works that guarantees 90% of all reads will be satisfied from a single sstable.

From DataStax Doc:

new sstables are added to the first level, L0, and immediately compacted with the sstables in L1. When L1 fills up, extra sstables are promoted to L2. Subsequent sstables generated in L1 will be compacted with the sstables in L2 with which they overlap.

解决方案

LeveledCompactionStrategy (LCS) in Cassandra implements the internals of LevelDB. You can check the exact implementation details in LevelDB implementation doc.

In order to give you a simple explanation take into account the following points:

1. Every sstable is created when a fixed (relatively small) size limit is reached. By default L0 gets 5MB files of files, and each subsequent level is 10x the size. (in L1 you'll have 50MB of data, L2 500MB, and so on).
2. Sstables are created with the guarantee that they don't overlap
3. When a level fills up, a compaction is triggered and stables from level-L are promoted to level-L+1. So, in L1 you'll have 50MB in ~10 files, L2 500MB in ~100 files, etc..

In bold are the relevant details that justify the 90% reads from the same file (sstable). Let's do the math together and everything will become clearer (I hope :)

Imagine you have keys A,B,C,D,E in L0, and each keys takes 1MB of data.

Next we insert key F. Because level 0 is filled a compaction will create a file with [A,B,C,D,E] in level 1, and F will remain in level 0.

That's ~83% of data in 1 file in L1

Next we insert G,H,I,J and K. So L0 fills up again, L1 gets a new sstable with [I,G,H,I,J].

By now we have K in L0, [A,B,C,D,E] and [I,G,H,I,J] in L1

And that's ~90% of data in L1 :)

If we continue inserting keys we will get around the same behavior so, that's why you get 90% of reads served from roughly the same file/sstable.

A more in depth and detailed (what happens with updates and tombstones) info is given in this paragraph on the link I mentioned (the sizes for compaction election are different because they are LevelDB defaults, not C*s):

When the size of level L exceeds its limit, we compact it in a background thread. The compaction picks a file from level L and all overlapping files from the next level L+1. Note that if a level-L file overlaps only part of a level-(L+1) file, the entire file at level-(L+1) is used as an input to the compaction and will be discarded after the compaction. Aside: because level-0 is special (files in it may overlap each other), we treat compactions from level-0 to level-1 specially: a level-0 compaction may pick more than one level-0 file in case some of these files overlap each other.

A compaction merges the contents of the picked files to produce a sequence of level-(L+1) files. We switch to producing a new level-(L+1) file after the current output file has reached the target file size (2MB). We also switch to a new output file when the key range of the current output file has grown enough to overlap more then ten level-(L+2) files. This last rule ensures that a later compaction of a level-(L+1) file will not pick up too much data from level-(L+2).

Hope this helps!

这篇关于分层压缩策略如何确保90％的读取来自一个稳定状态的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！