Python redis subscribe 无法获取所有数据? [英] Python redis subscribe can not get all datas?

问题描述

我正在使用 python 从 redis 获取数据，然后将其解析为 kafka.它在大多数情况下都能很好地工作.

I am using python to get datas from redis and then parser it to kafka. It works well in most situations.

但是当我用python模拟构建数据到redis时，或者queuen中有快速放入数据时，我无法获取所有数据.

But when I use python to simulate build the data to redis, or there was a fast put-in datas in queuen, I can't get all the data.

这是我关于redis生产者模拟构建20000个数据到redis的代码:

Here is my code about redis producer to simulate build 20000 datas to redis:

rc = redis.Redis(host='127.0.0.1', port=6379)
rc.ping()
ps = rc.pubsub()
ps.subscribe('bdwaf')
r_str = "--8198b507-A--\n[22/Jun/2017:14:13:19 +0800]ucTcxcMcicAcAcAcicAcAcAm 192.168.1.189 50054 127.0.0.1 80\n"
for i in range(0, 20000):
    rc.publish('bdwaf', r_str)

redis 消费者也是 kafka 生产者:

and the redis consumer also the kafka producer is:

rc = redis.Redis(host='localhost', port=6379)
rc.ping()
ps = rc.pubsub()
ps.subscribe('bdwaf')
num = 0
for item in ps.listen():
     if item['type'] == 'message':
         num += 1
         a.parser(item['data'])
         print num

它只打印出 4000 条数据.

It only prints out like 4000 datas.

如果我注释a.parser(item['data'])，它可以打印出所有数据编号.

If I comment the a.parser(item['data']), it can print out all datas num.

或者在redis生产者中写入sleep(0.001)，也可以打印出所有的数据号.

Or wirte sleep(0.001) in the redis producer, it can print out all datas num too.

我的代码有什么问题?

推荐答案

我假设您正在使用 redis-呸.

文档将listen 称为旧版本的lib...也许您应该使用另一种方法来读取消息.例如带有回调

The documentation refers to listen as older version of the lib... Maybe you should use another method for message reading. For example with a callback

p = r.pubsub()

def my_handler(message):
    print 'MY HANDLER: ', message['data']
    if item['type'] == 'message':
         num += 1
         a.parser(item['data'])
         print num

p.subscribe('bdwaf', my_handler)
# read the subscribe confirmation message
p.get_message()

当您一次发布 20000 条消息时，您的 redis 服务器可能会耗尽内存.尝试在 redis.conf 文件中增加 redis 内存

It is possible that your redis server is running out of memory when you publish 20000 messages at once. Try increasing redis memory in redis.conf file

maxmemory 500mb # or greater if needed

是内存问题，查看this 问题以获取有关如何处理它的更多信息.

It is a memory problem, check out this question for more information on how to handle it.

这篇关于Python redis subscribe 无法获取所有数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！