redis + gevent - 性能不佳 - 我做错了什么? [英] redis + gevent - Poor performance - what am I doing wrong?

问题描述

我刚刚编写了一段简单的代码来对 Redis + gevent 进行性能测试，以了解异步如何帮助提高性能，但我惊讶地发现性能不佳.这是我的代码.如果你去掉前两行来修补这段代码，那么你将看到正常执行"时间.

I just wrote a simple piece of code to perf test Redis + gevent to see how async helps perforamance and I was surprised to find bad performance. here is my code. If you get rid of the first two lines to monkey patch this code then you will see the "normal execution" timing.

在 Ubuntu 12.04 LTS 虚拟机上，我看到时间为

On a Ubuntu 12.04 LTS VM, I am seeing a timing of

没有猴子补丁 - 54 秒使用猴子补丁 - 61 秒

without monkey patch - 54 secs With monkey patch - 61 seconds

我的代码/方法有问题吗?这里有性能问题吗?

Is there something wrong with my code / approach? Is there a perf issue here?

#!/usr/bin/python

from gevent import monkey

monkey.patch_all()

import timeit
import redis
from redis.connection import UnixDomainSocketConnection

def UxDomainSocket():
    pool = redis.ConnectionPool(connection_class=UnixDomainSocketConnection, path =    '/var/redis/redis.sock')
    r = redis.Redis(connection_pool = pool)
    r.set("testsocket", 1)
    for i in range(100):
            r.incr('testsocket', 10)
    r.get('testsocket')
    r.delete('testsocket')


print timeit.Timer(stmt='UxDomainSocket()',
 setup='from __main__ import UxDomainSocket').timeit(number=1000)

推荐答案

这是意料之中的.

您在系统调用成本高于物理硬件的虚拟机上运行此基准测试.当 gevent 被激活时，它往往会生成更多的系统调用(以处理 epoll 设备)，因此最终会降低性能.

You run this benchmark on a VM, on which the cost of system calls is higher than on physical hardware. When gevent is activated, it tends to generate more system calls (to handle the epoll device), so you end up with less performance.

您可以通过在脚本上使用 strace 轻松检查这一点.

You can easily check this point by using strace on the script.

没有gevent，内循环生成:

Without gevent, the inner loop generates:

recvfrom(3, ":931
", 4096, 0, NULL, NULL) = 6
sendto(3, "*3
$6
INCRBY
$10
testsocket
"..., 41, 0, NULL, 0) = 41
recvfrom(3, ":941
", 4096, 0, NULL, NULL) = 6
sendto(3, "*3
$6
INCRBY
$10
testsocket
"..., 41, 0, NULL, 0) = 41

使用 gevent，您将遇到以下情况:

With gevent, you will have occurences of:

recvfrom(3, ":221
", 4096, 0, NULL, NULL) = 6
sendto(3, "*3
$6
INCRBY
$10
testsocket
"..., 41, 0, NULL, 0) = 41
recvfrom(3, 0x7b0f04, 4096, 0, 0, 0)    = -1 EAGAIN (Resource temporarily unavailable)
epoll_ctl(5, EPOLL_CTL_ADD, 3, {EPOLLIN, {u32=3, u64=3}}) = 0
epoll_wait(5, {{EPOLLIN, {u32=3, u64=3}}}, 32, 4294967295) = 1
clock_gettime(CLOCK_MONOTONIC, {2469, 779710323}) = 0
epoll_ctl(5, EPOLL_CTL_DEL, 3, {EPOLLIN, {u32=3, u64=3}}) = 0
recvfrom(3, ":231
", 4096, 0, NULL, NULL) = 6
sendto(3, "*3
$6
INCRBY
$10
testsocket
"..., 41, 0, NULL, 0) = 41

当 recvfrom 调用阻塞 (EAGAIN) 时，gevent 返回到事件循环，因此会执行额外的调用以等待文件描述符事件 (epoll_wait).

When the recvfrom call is blocking (EAGAIN), gevent goes back to the event loop, so additional calls are done to wait for file descriptors events (epoll_wait).

请注意，这种基准测试对于任何事件循环系统来说都是最糟糕的情况，因为您只有一个文件描述符，因此无法在多个描述符上分解等待操作.此外，异步 I/O 在这里无法改进任何东西，因为一切都是同步的.

Please note this kind of benchmark is a worst case for any event loop system, because you only have one file descriptor, so the wait operations cannot be factorized on several descriptors. Furthermore, async I/Os cannot improve anything here since everything is synchronous.

这也是Redis最坏的情况，因为:

It is also a worst case for Redis because:

• 它会生成许多到服务器的往返

• it generates many roundtrips to the server

它系统地连接/断开连接(1000 次)，因为池是在 UxDomainSocket 函数中声明的.

it systematically connects/disconnects (1000 times) because the pool is declared in UxDomainSocket function.

实际上，您的基准测试并未测试 gevent、redis 或 redis-py:它锻炼了 VM 的能力，以维持两个进程之间的乒乓游戏.

Actually your benchmark does not test gevent, redis or redis-py: it exercises the capability of a VM to sustain a ping-pong game between 2 processes.

如果你想提高性能，你需要:

If you want to increase performance, you need to:

• 使用流水线来减少往返次数

• use pipelining to decrease the number of roundtrips

使池在整个基准测试中保持持久

make the pool persistent across the whole benchmark

例如，考虑使用以下脚本:

For instance, consider with the following script:

#!/usr/bin/python

from gevent import monkey
monkey.patch_all()

import timeit
import redis
from redis.connection import UnixDomainSocketConnection

pool = redis.ConnectionPool(connection_class=UnixDomainSocketConnection, path = '/tmp/redis.sock')

def UxDomainSocket():
    r = redis.Redis(connection_pool = pool)
    p = r.pipeline(transaction=False)
    p.set("testsocket", 1)
    for i in range(100):
        p.incr('testsocket', 10)
    p.get('testsocket')
    p.delete('testsocket')
    p.execute()

print timeit.Timer(stmt='UxDomainSocket()', setup='from __main__ import UxDomainSocket').timeit(number=1000)

使用这个脚本，我获得了大约 3 倍的性能提升，并且几乎没有 gevent 的开销.

With this script, I get about 3x better performance and almost no overhead with gevent.

这篇关于redis + gevent - 性能不佳 - 我做错了什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！