redis + gevent-效果不佳-我在做什么错? [英] redis + gevent - Poor performance - what am I doing wrong?

问题描述

我只是编写了一段简单的代码来对Redis + gevent进行性能测试，以查看异步如何帮助性能，而令我惊讶的是发现性能不佳.这是我的代码.如果您摆脱了前两行猴子打补丁的代码，那么您将看到正常执行"的时间.

I just wrote a simple piece of code to perf test Redis + gevent to see how async helps perforamance and I was surprised to find bad performance. here is my code. If you get rid of the first two lines to monkey patch this code then you will see the "normal execution" timing.

在Ubuntu 12.04 LTS VM上，我看到的时间是

On a Ubuntu 12.04 LTS VM, I am seeing a timing of

没有猴子补丁-54秒 有猴子补丁-61秒

without monkey patch - 54 secs With monkey patch - 61 seconds

我的代码/方法有问题吗?这里有性能问题吗?

Is there something wrong with my code / approach? Is there a perf issue here?

#!/usr/bin/python

from gevent import monkey

monkey.patch_all()

import timeit
import redis
from redis.connection import UnixDomainSocketConnection

def UxDomainSocket():
    pool = redis.ConnectionPool(connection_class=UnixDomainSocketConnection, path =    '/var/redis/redis.sock')
    r = redis.Redis(connection_pool = pool)
    r.set("testsocket", 1)
    for i in range(100):
            r.incr('testsocket', 10)
    r.get('testsocket')
    r.delete('testsocket')


print timeit.Timer(stmt='UxDomainSocket()',
 setup='from __main__ import UxDomainSocket').timeit(number=1000)

推荐答案

这是预期的结果.

您在VM上运行此基准测试，在该VM上系统调用的成本高于物理硬件上的成本.激活gevent时，它往往会生成更多的系统调用(以处理epoll设备)，因此最终会降低性能.

You run this benchmark on a VM, on which the cost of system calls is higher than on physical hardware. When gevent is activated, it tends to generate more system calls (to handle the epoll device), so you end up with less performance.

您可以通过在脚本上使用strace轻松地检查这一点.

You can easily check this point by using strace on the script.

没有gevent，内部循环会生成:

Without gevent, the inner loop generates:

recvfrom(3, ":931\r\n", 4096, 0, NULL, NULL) = 6
sendto(3, "*3\r\n$6\r\nINCRBY\r\n$10\r\ntestsocket\r"..., 41, 0, NULL, 0) = 41
recvfrom(3, ":941\r\n", 4096, 0, NULL, NULL) = 6
sendto(3, "*3\r\n$6\r\nINCRBY\r\n$10\r\ntestsocket\r"..., 41, 0, NULL, 0) = 41

使用gevent，您将发生以下情况:

With gevent, you will have occurences of:

recvfrom(3, ":221\r\n", 4096, 0, NULL, NULL) = 6
sendto(3, "*3\r\n$6\r\nINCRBY\r\n$10\r\ntestsocket\r"..., 41, 0, NULL, 0) = 41
recvfrom(3, 0x7b0f04, 4096, 0, 0, 0)    = -1 EAGAIN (Resource temporarily unavailable)
epoll_ctl(5, EPOLL_CTL_ADD, 3, {EPOLLIN, {u32=3, u64=3}}) = 0
epoll_wait(5, {{EPOLLIN, {u32=3, u64=3}}}, 32, 4294967295) = 1
clock_gettime(CLOCK_MONOTONIC, {2469, 779710323}) = 0
epoll_ctl(5, EPOLL_CTL_DEL, 3, {EPOLLIN, {u32=3, u64=3}}) = 0
recvfrom(3, ":231\r\n", 4096, 0, NULL, NULL) = 6
sendto(3, "*3\r\n$6\r\nINCRBY\r\n$10\r\ntestsocket\r"..., 41, 0, NULL, 0) = 41

当recvfrom调用被阻止(EAGAIN)时，gevent返回事件循环，因此需要执行其他调用来等待文件描述符事件(epoll_wait).

When the recvfrom call is blocking (EAGAIN), gevent goes back to the event loop, so additional calls are done to wait for file descriptors events (epoll_wait).

请注意，对于任何事件循环系统，这种基准测试都是最坏的情况，因为您只有一个文件描述符，因此无法在多个描述符上分解等待操作.此外，由于所有内容都是同步的，因此异步I/O在这里无法改善任何情况.

Please note this kind of benchmark is a worst case for any event loop system, because you only have one file descriptor, so the wait operations cannot be factorized on several descriptors. Furthermore, async I/Os cannot improve anything here since everything is synchronous.

对于Redis来说，这也是最糟糕的情况，因为:

It is also a worst case for Redis because:

• 它会生成许多到服务器的往返行程

• it generates many roundtrips to the server

它系统地连接/断开连接(1000次)，因为该池是在UxDomainSocket函数中声明的.

it systematically connects/disconnects (1000 times) because the pool is declared in UxDomainSocket function.

实际上，您的基准测试不会测试gevent，redis或redis-py:它可以行使VM在2个进程之间维持乒乓游戏的能力.

Actually your benchmark does not test gevent, redis or redis-py: it exercises the capability of a VM to sustain a ping-pong game between 2 processes.

如果要提高性能，则需要:

If you want to increase performance, you need to:

• 使用流水线减少往返次数

• use pipelining to decrease the number of roundtrips

使池在整个基准测试中持久存在

make the pool persistent across the whole benchmark

例如，考虑以下脚本:

#!/usr/bin/python

from gevent import monkey
monkey.patch_all()

import timeit
import redis
from redis.connection import UnixDomainSocketConnection

pool = redis.ConnectionPool(connection_class=UnixDomainSocketConnection, path = '/tmp/redis.sock')

def UxDomainSocket():
    r = redis.Redis(connection_pool = pool)
    p = r.pipeline(transaction=False)
    p.set("testsocket", 1)
    for i in range(100):
        p.incr('testsocket', 10)
    p.get('testsocket')
    p.delete('testsocket')
    p.execute()

print timeit.Timer(stmt='UxDomainSocket()', setup='from __main__ import UxDomainSocket').timeit(number=1000)

使用此脚本，我的性能提高了约3倍，而使用gevent几乎没有开销.

With this script, I get about 3x better performance and almost no overhead with gevent.

这篇关于redis + gevent-效果不佳-我在做什么错?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！