如何提高Redis服务器的CPU使用率? [英] How to improve Redis server's CPU usage?

查看：104
本文介绍了如何提高Redis服务器的CPU使用率?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我的目标是使我们的Redis服务器在生产中达到约80％的CPU利用率.通过确保我们不会利用CPU不足，同时为增长和高峰留出一些空间，这将有益于后端服务器设计.

My goal is for our Redis server to hit about 80% CPU utilization in production. This will benefit our backend server design by ensuring we are not under-utilizing CPU while also leaving some headroom for growth and spikes.

使用Redis自己的基准测试工具redis-benchmark时，很容易达到大约100％的CPU使用率:

While using Redis' own benchmark tool redis-benchmark, it's very easy to reach about 100% CPU usage:

    $ redis-benchmark -h 192.168.1.6 -n 1000000 -c 50

在此基准测试中，我们分配了50个客户端以将1,000,000个请求推送到我们的Redis服务器.

On this benchmark, we assigned 50 clients to push 1,000,000 requests to our Redis server.

但是在使用其他一些客户端工具(例如 redis-lua 或 webdis )，最大CPU使用率不到60％.

But while using some other client tools(such as redis-lua or webdis), the maximum CPU usage is less than 60%.

我浏览了webdis和redis-lua中的一些代码. webdis依赖于hiredis，redis-lua是在Lua中实现的，并且依赖于socket(lua-socket).

I browsed some code in webdis and redis-lua. webdis depends on hiredis, and redis-lua is implemented in Lua, and depends on socket(lua-socket).

与Redis基准相比，这些客户端是否太慢，并且无法最大化Redis CPU的使用量?

Are these clients too slow compared to the Redis benchmark and are unable to maximize the Redis CPU consumption?

我还浏览了redis-benchmark.c中的一些代码.基准测试的主要工作在aeMain中完成.似乎redis-benchmark使用Redis的快速代码，而我的测试客户端(webdis和redis-lua)却没有.

I also browsed some code in redis-benchmark.c. The benchmark's main work is done in aeMain. It seems like that the redis-benchmark uses fast code from Redis, and my test clients(webdis and redis-lua) do not.

目前我的客户有两种选择:

Currently my client have two choices:

  1. 使用redis-lua
  2. 使用webdis
    3. 之类的工具
  1. Use redis-lua
  2. Use a tool like webdis

但是，这两个不能使Redis的CPU使用率最大化(小于60％).还有其他选择吗?

However, these two do not maximize Redis' CPU utilization (less than 60%). Are there any other choices?

或者，是否有可能在redis-benchmark工具本身之外充分利用redis-server?

Or, is it possible to fully utilize redis-server outside of the redis-benchmark tool itself?

推荐答案

我怀疑最大限度地提高Redis的CPU使用率会有益于您的后端设计.正确的问题是Redis是否足以有效地维持给定延迟下的吞吐量. Redis是单线程服务器:CPU消耗80％时，延迟可能非常糟糕.

I doubt maximizing CPU usage of Redis will benefit your backend design. The right question is rather whether Redis is efficient enough to sustain your throughput at a given latency. Redis is a single-threaded server: at 80% CPU consumption, the latency will likely be very bad.

我建议您在尝试增加Redis CPU消耗之前，在redis-benchmark工作期间测量延迟，以查看它是否可以满足您的需求. redis-cli的--latency选项可用于此目的:

I suggest you measure latency while redis-benchmark is working to see if it is acceptable for your needs before trying to increase Redis CPU consumption. The --latency option of redis-cli can be used for this:

  • 启动redis服务器
  • 尝试redis-cli --latency，记下avg值，将其停止
  • 在另一个窗口中，启动基准测试，并确保其运行一段时间
  • 尝试redis-cli --latency，记下avg值，将其停止
  • 停止基准测试
  • 比较两个avg值

现在，如果您确实想增加Redis CPU的使用量，则需要一个高效的客户端程序(例如redis-benchmark)，该客户端程序能够同时处理多个连接，或者是客户端程序的多个实例.

Now, if you really want to increase Redis CPU consumption, you need either an efficient client program (like redis-benchmark), able to handle multiple connections concurrently, either multiple instances of your client program.

Lua是一种快速的解释语言，但它仍然是一种解释语言.这将比C代码慢一两个数量级. Redis在解析/生成其协议方面比lua-redis快得多，因此您将无法使用唯一的Lua客户端使Redis饱和(除非您使用O(n)Redis命令-稍后请参见).

Lua is a fast interpreted language, but it is still an interpreted language. It will be one or two orders of magnitude slower than C code. Redis is much faster at parsing/generating its protocol than lua-redis, so you will not be able to saturate Redis with a unique Lua client (except if you use O(n) Redis commands - see later).

webdis用高效的客户端库用C实现，但是必须解析http/json协议，该协议比Redis协议更加冗长和复杂.在大多数操作中，它可能比Redis本身消耗更多的CPU.再次重申，您不会使用单个webdis实例饱和Redis.

webdis is implemented in C, with an efficient client library, but has to parse the http/json protocols which happen to be more verbose and complex than Redis protocol. It likely consumes more CPU than Redis itself for most operations. So again, you will not saturate Redis with a single webdis instance.

下面是一些使Redis与多个Lua客户端饱和的示例.

Here are some examples to saturate Redis with multiple Lua clients.

如果尚未完成，建议您先查看 Redis基准页面.

If it is not already done, I suggest you had a look at the Redis benchmark page first.

如果您在与Redis相同的框中运行基准测试:

关键点是将一个内核专用于Redis，并在其他内核上运行客户端程序.在Linux上，您可以为此使用tasket命令.

The key point is to dedicate a core to Redis, and run the client programs on the other cores. On Linux, you can use the taskset command for this.

# Start Redis on core 0
taskset -c 0 redis-server redis.conf

# Start Lua programs on the other cores
for x in `seq 1 10` ; do taskset -c 1,2,3 luajit example.lua & done

Lua程序应使用流水线以最大化吞吐量并减少系统活动.

The Lua program should use pipelining to maximize throughput and reduce system activity.

local redis = require 'redis'
local client = redis.connect('127.0.0.1', 6379)
for i=1,1000000 do
    local replies = client:pipeline(function(p)
    for j=1,1000 do
            local key = 'counter:'..tostring(j)
            p:incrby(key,1)
        end
    end)
end

在我的系统上，Lua程序占用的Redis CPU数量是Redis的4倍以上，因此您需要使用4个以上的内核来使Redis达到饱和(6核的盒子应该没问题).

On my system, the Lua program takes more than 4 times the CPU of Redis, so you need more than 4 cores to saturate Redis with this method (a 6 cores box should be fine).

如果您在与Redis不同的机器上运行基准测试

除非您在CPU不足的虚拟机上运行，​​否则在这种情况下，瓶颈可能是网络.我不认为您可以使用小于1 GbE的链接来使Redis饱和.

Except if you run on CPU starved virtual machines, the bottleneck will likely be the network in that case. I don't think you can saturate Redis with anything less than a 1 GbE link.

请确保尽可能地对查询进行管线化(请参见前面的Lua程序)，以避免网络延迟瓶颈，并减少CPU上网络中断的成本(填充以太网数据包).尝试在未绑定到网卡的核心上运行Redis(并处理网络中断).您可以使用htop之类的工具来检查这一点.

Be sure to pipeline your queries as far as you can (see the previous Lua program) to avoid the network latency bottleneck, and reduce the cost of network interrupts on the CPU (filling ethernet packets). Try to run Redis on a core which is not bound to the network card (and processes network interrupts). You can use tools like htop to check this last point.

如果可以，请尝试在网络的其他各种机器上运行Lua客户端.同样，您将需要大量Lua客户端来使Redis饱和(6-10应该可以).

Try to run your Lua clients on various other machines of the network if you can. Again you will need a good number of Lua clients to saturate Redis (6-10 should be fine).

在某些情况下，唯一的Lua流程就足够了:

现在，如果每个查询足够昂贵，则可以使用单个Lua客户端使Redis饱和.这是一个示例:

Now, it is possible to saturate Redis with a single Lua client if each query is expensive enough. Here is an example:

local redis = require 'redis'
local client = redis.connect('127.0.0.1', 6379)

for i=1,1000 do
    local replies = client:pipeline(function(p)
        for j=1,1000 do
            p:rpush("toto",i*1000+j)
        end
    end)
end

N = 500000
for i=1,100000 do
    local replies = client:pipeline(function(p)
        for j=1,10 do
            p:lrange("toto",N, N+10)
        end
    end)
end

此程序将在列表中填充100万个项目，然后使用lrange命令从列表中间获取10个项目(Redis的最坏情况).因此，每次执行查询时，服务器都会扫描500K项.因为仅返回10个项目，所以它们可以通过lua-redis进行快速解析，而不会消耗CPU.在这种情况下，所有CPU消耗都将在服务器端.

This program populates a list with 1M items, and then uses lrange commands to fetch 10 items from the middle of the list (worst case for Redis). So each time a query is executed, 500K items are scanned by the server. Because only 10 items are returned, they are fast to parse by lua-redis which will not consume CPU. In this situation, all the CPU consumption will be on server side.

最后一句话

Redis客户端可能比redis-lua更快:

There are probably faster Redis clients than redis-lua:

  • https://github.com/agladysh/lua-hiredis (based on hiredis)
  • https://github.com/agladysh/ljffi-hiredis (based on hiredis, using luajit FFI)

您可能想尝试一下.

这篇关于如何提高Redis服务器的CPU使用率?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆