配置redis以始终先驱逐旧数据 [英] Configuring redis to consistently evict older data first

问题描述

我正在Redis中存储一堆实时数据。我为所有按键设置了14400秒（4小时）的TTL。我已将maxmemory设置为10G，当前该空间不足以容纳4个小时的数据，并且我没有使用虚拟内存，因此redis会在数据过期之前逐出数据。

I'm storing a bunch of realtime data in redis. I'm setting a TTL of 14400 seconds (4 hours) on all of the keys. I've set maxmemory to 10G, which currently is not enough space to fit 4 hours of data in memory, and I'm not using virtual memory, so redis is evicting data before it expires.

我可以撤消数据，但是我希望它首先移出最旧的数据。因此，即使我没有完整的4个小时的数据，至少我也可以拥有一定范围的数据（3个小时，2个小时等），没有任何差距。我试图通过设置 maxmemory-policy = volatile-ttl 来实现这一点，认为最旧的键将首先被驱逐，因为它们都具有相同的TTL，但是这样做不起作用办法。看来redis任意地驱逐了数据，所以最终我的数据出现了空白。例如，今天从2012-01-25T13：00开始的数据在2012-01-25T12：00之前被逐出。

I'm okay with redis evicting the data, but I would like it to evict the oldest data first. So even if I don't have a full 4 hours of data, at least I can have some range of data (3 hours, 2 hours, etc) with no gaps in it. I tried to accomplish this by setting maxmemory-policy=volatile-ttl, thinking that the oldest keys would be evicted first since they all have the same TTL, but it's not working that way. It appears that redis is evicting data somewhat arbitrarily, so I end up with gaps in my data. For example, today the data from 2012-01-25T13:00 was evicted before the data from 2012-01-25T12:00.

是否可以将redis配置为始终先删除旧数据？

Is it possible to configure redis to consistently evict the older data first?

以下是我的redis.cnf文件中的相关行。让我知道是否要查看其他配置：

Here are the relevant lines from my redis.cnf file. Let me know if you want to see any more of the cofiguration:

maxmemory 10gb
maxmemory-policy volatile-ttl
vm-enabled no

推荐答案

AFAIK，无法配置Redis首先一致地逐出旧数据。

AFAIK, it is not possible to configure Redis to consistently evict the older data first.

在maxmemory-policy中选择* -ttl或* -lru选项时，Redis不使用选择要删除的密钥的精确算法。一个精确的算法需要在内存中有一个额外的列表（用于* -lru）或一个额外的堆（对于* -ttl），并将其与常规Redis字典数据结构进行交叉引用。就内存消耗而言，这将是昂贵的。

When the *-ttl or *-lru options are chosen in maxmemory-policy, Redis does not use an exact algorithm to pick the keys to be removed. An exact algorithm would require an extra list (for *-lru) or an extra heap (for *-ttl) in memory, and cross-reference it with the normal Redis dictionary data structure. It would be expensive in term of memory consumption.

使用当前机制，逐出发生在主事件循环中（即，在每个命令之前的每次循环迭代中检查潜在的逐出）被执行）。在内存返回到最大内存限制以下之前，Redis会随机选择n个密钥的样本，并选择最空闲的密钥（对于* -lru）或最接近其到期限制的密钥（对于* -ttl）进行过期。默认情况下，仅考虑3个样本。结果是不确定的。

With the current mechanism, evictions occur in the main event loop (i.e. potential evictions are checked at each loop iteration before each command is executed). Until memory is back under the maxmemory limit, Redis randomly picks a sample of n keys, and selects for expiration the most idle one (for *-lru) or the one which is the closest to its expiration limit (for *-ttl). By default only 3 samples are considered. The result is non deterministic.

提高此算法准确性并缓解问题的一种方法是增加考虑的样本数（配置中的maxmemory-samples参数文件）。
请勿将其设置得太高，因为它将消耗一些CPU。这是逐出准确性与CPU消耗之间的折衷。

One way to increase the accuracy of this algorithm and mitigate the problem is to increase the number of considered samples (maxmemory-samples parameter in the configuration file). Do not set it too high, since it will consume some CPU. It is a tradeoff between eviction accuracy and CPU consumption.

现在，如果您确实需要一致的行为，一种解决方案是在Redis之上实现自己的逐出机制。例如，您可以添加一个列表（用于不可更新的密钥）或一个已排序的集合（用于可更新的密钥），以便跟踪应首先收回的密钥。然后，添加一个守护程序，该守护程序的目的是定期（使用INFO）检查内存消耗，并查询列表/排序集中的项以删除相关的键。

Now if you really require a consistent behavior, one solution is to implement your own eviction mechanism on top of Redis. For instance, you could add a list (for non updatable keys) or a sorted set (for updatable keys) in order to track the keys that should be evicted first. Then, you add a daemon whose purpose is to periodically check (using INFO) the memory consumption and query the items of the list/sorted set to remove the relevant keys.

请注意，其他缓存系统也有自己的方式来处理此问题。例如，对于memcached，每个平板都有一个LRU结构（取决于对象大小），因此逐出顺序也不准确（尽管实际上比Redis更具确定性）。

Please note other caching systems have their own way to deal with this problem. For instance with memcached, there is one LRU structure per slab (which depends on the object size), so the eviction order is also not accurate (although more deterministic than with Redis in practice).

这篇关于配置redis以始终先驱逐旧数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！