在 redis db 中迭代所有键和值的更快方法 [英] Faster way to iterate all keys and values in redis db

问题描述

我有一个大约有 350,000 个键的数据库.目前我的代码只是遍历所有键并从数据库中获取其值.

I have a db with about 350,000 keys. Currently my code just loops through all keys and gets its value from the db.

然而这需要将近 2 分钟才能完成，这看起来很慢，redis-benchmark 给出了 100k reqs/3s.

However this takes almost 2 minutes to do, which seems really slow, redis-benchmark gave 100k reqs/3s.

我已经研究了流水线，但我需要返回每个值，以便我最终得到一个键值对的字典.

I've looked at pipelining but I need each value returned so that I end up with a dict of key, value pairs.

目前，如果可能的话，我正在考虑在我的代码中使用线程来加快速度，这是处理这个用例的最佳方式吗?

At the moment I'm thinking of using threading in my code if possible to speed this up, is this the best way to handle this usecase?

这是我目前的代码.

import redis, timeit
start_time = timeit.default_timer()
count = redis.Redis(host='127.0.0.1', port=6379, db=9)
keys = count.keys()

data = {}

for key in keys:
    value = count.get(key)
    if value:
        data[key.decode('utf-8')] = int(value.decode('utf-8'))

elapsed = timeit.default_timer() - start_time

print('Time to read {} records: '.format(len(keys)), elapsed)

推荐答案

首先，最快的方法是在 评估.

First, the fastest way is doing all of this inside EVAL.

接下来，推荐的迭代所有键的方法是SCAN.它的迭代速度不会比 KEYS 快，但允许 Redis 在两者之间处理一些其他操作，因此它将有助于整体应用程序行为.

Next, recommended approach to iterate all keys is SCAN. It would not iterate faster than KEYS, but will allow Redis to process some other actions in between, so it will help with overall application behavior.

脚本类似于 local data={} local i=1 local mykeys=redis.call(\"KEYS\",\"*\") for k=1,#mykeys do local tmpkey=mykeys[k] data[i]={tmpkey,redis.call(\"GET\",tmpkey)} i=i+1 结束返回数据，但是如果你有无法通过 GET 访问的键，它会失败(如集合、列表).您需要为其添加错误处理.如果需要排序，可以直接在 LUA 中进行，也可以稍后在客户端进行.第二个会比较慢，但不会让其他redis实例的用户等待.

The script will be something like local data={} local i=1 local mykeys=redis.call(\"KEYS\",\"*\") for k=1,#mykeys do local tmpkey=mykeys[k] data[i]={tmpkey,redis.call(\"GET\",tmpkey)} i=i+1 end return data, but it will fail if you have keys inaccessible with GET (like sets, lists). You need to add error handling to it. If you need sorting, you can do it either in LUA directly, or later on the client side. The second will be slower, but would not let other users of redis instance wait.

示例输出:

127.0.0.1:6370> eval "local data={} local i=1 local mykeys=redis.call(\"KEYS\",\"*\") for k=1,#mykeys do local tmpkey=mykeys[k] data[i]={tmpkey,redis.call(\"GET\",tmpkey)} i=i+1 end return data" 0
1) 1) "a"
   2) "aval"
2) 1) "b"
   2) "bval"
3) 1) "c"
   2) "cval"
4) 1) "d"
   2) "dval"
5) 1) "e"
   2) "eval"
6) 1) "f"
   2) "fval"
7) 1) "g"
   2) "gval"
8) 1) "h"
   2) "hval"

这篇关于在 redis db 中迭代所有键和值的更快方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！