Python Redis订阅无法获取所有数据? [英] Python redis subscribe can not get all datas?

问题描述

我正在使用python从redis获取数据，然后将其解析为kafka.在大多数情况下效果很好.

I am using python to get datas from redis and then parser it to kafka. It works well in most situations.

但是当我使用python模拟构建数据以进行redis时，或者队列中有快速放入的数据时，我无法获取所有数据.

But when I use python to simulate build the data to redis, or there was a fast put-in datas in queuen, I can't get all the data.

这是我的有关redis生产者的代码，用于模拟内部版本20000数据到redis:

Here is my code about redis producer to simulate build 20000 datas to redis:

rc = redis.Redis(host='127.0.0.1', port=6379)
rc.ping()
ps = rc.pubsub()
ps.subscribe('bdwaf')
r_str = "--8198b507-A--\n[22/Jun/2017:14:13:19 +0800]ucTcxcMcicAcAcAcicAcAcAm 192.168.1.189 50054 127.0.0.1 80\n"
for i in range(0, 20000):
    rc.publish('bdwaf', r_str)

redis消费者也是kafka生产者:

and the redis consumer also the kafka producer is:

rc = redis.Redis(host='localhost', port=6379)
rc.ping()
ps = rc.pubsub()
ps.subscribe('bdwaf')
num = 0
for item in ps.listen():
     if item['type'] == 'message':
         num += 1
         a.parser(item['data'])
         print num

它只能打印出4000条数据.

It only prints out like 4000 datas.

如果我评论 a.parser(item ['data'])，它可以打印出所有数据num.

If I comment the a.parser(item['data']), it can print out all datas num.

或者在redis生产者中编写 sleep(0.001)，它也可以打印出所有数据num.

Or wirte sleep(0.001) in the redis producer, it can print out all datas num too.

我的代码有什么问题?

推荐答案

我假设您正在使用 redis-py .

文档将 listen 称为lib的较早版本.也许您应该使用另一种方法来读取消息.例如带有回调

The documentation refers to listen as older version of the lib... Maybe you should use another method for message reading. For example with a callback

p = r.pubsub()

def my_handler(message):
    print 'MY HANDLER: ', message['data']
    if item['type'] == 'message':
         num += 1
         a.parser(item['data'])
         print num

p.subscribe('bdwaf', my_handler)
# read the subscribe confirmation message
p.get_message()

一次发布20000条消息时，redis服务器可能内存不足.尝试增加 redis.conf 文件

It is possible that your redis server is running out of memory when you publish 20000 messages at once. Try increasing redis memory in redis.conf file

maxmemory 500mb # or greater if needed

这是内存问题，请查看

It is a memory problem, check out this question for more information on how to handle it.

这篇关于Python Redis订阅无法获取所有数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！