批量摄取到 Redis [英] Bulk ingest into Redis

问题描述

我正在尝试尽快将大量数据加载到 Redis 中.

I'm trying to load a large piece of data into Redis as fast as possible.

我的数据如下:

771240491921 SOME;STRING;ABOUT;THIS;LENGTH
345928354912 SOME;STRING;ABOUT;THIS;LENGTH

左边有一个 ~12 位数字，右边有一个可变长度的字符串.键是左边的数字，数据是右边的字符串.

There is a ~12 digit number on the left and a variable length string on the right. The key is going to be the number on the left and the data is going to be the string on the right.

在我开箱即用的 Redis 实例中，使用包含这些数据的未压缩纯文本文件，我可以在一分钟内获取大约一百万条记录.我需要做大约 4500 万次，大约需要 45 分钟.45 分钟太长了.

In my Redis instance that I just installed out of the box and with an uncompressed plain text file with this data, I can get about a million records into it a minute. I need to do about 45 million, which would take about 45 minutes. 45 minutes is too long.

是否有一些标准的性能调整可供我进行此类优化?通过在不同的实例之间进行分片，我会获得更好的性能吗?

Are there some standard performance tweaks that exist for me to do this type of optimization? Would I get better performance by sharding across separate instances?

推荐答案

最快的方法如下:从这些数据中生成 Redis 协议.Redis.io 站点上提供了生成 Redis 协议的文档，它是一个简单的协议.完成后，只需将其命名为 appendonly.log 并以仅附加模式启动 redis.

The fastest way to do this is the following: generate Redis protocol out of this data. The documentation to generate the Redis protocol is on the Redis.io site, it is a trivial protocol. Once you have that, just call it appendonly.log and start redis in append only mode.

您甚至可以执行 FLUSHALL 命令，最后使用 netcat 将数据推送到您的服务器，将输出重定向到/dev/null.

You can even do a FLUSHALL command and finally push the data into your server with netcat, redirecting the output to /dev/null.

这将是超快的，没有 RTT 等待，它只是大量加载数据.

This will be super fast, there is no RTT to wait, it's just a bulk loading of data.

较少的hackish方式，只需使用流水线每次插入1000个东西.它几乎和生成协议一样快，但更干净:)

Less hackish way, just insert things 1000 per time using pipelining. It's almost as fast as generating the protocol, but much more clean :)

这篇关于批量摄取到 Redis的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！