在Redis中存储Numpy数组的最快方法 [英] Fastest way to store a numpy array in redis

问题描述

我正在AI项目上使用Redis.

I'm using redis on an AI project.

这个想法是让多个环境模拟器在许多cpu内核上运行策略.模拟器将体验(状态/操作/奖励元组列表)写入Redis服务器(重播缓冲区).然后，培训过程将经验作为数据集读取以生成新策略.将新策略部署到模拟器，删除先前运行的数据，然后继续该过程.

The idea is to have multiple environment simulators running policies on a lot of cpu cores. The simulators write experience (a list of state/action/reward tuples) to a redis server (replay buffer). Then a training process reads the experience as a dataset to generate a new policy. New policy is deployed to the simulators, data from previous run is deleted, and the process continues.

大部分体验都记录在状态"中.通常将其表示为尺寸为80 x 80的大型numpy数组.模拟器会以cpu允许的最快速度生成它们.

The bulk of the experience is captured in the "state". Which is normally represented as a large numpy array of dimension say, 80 x 80. The simulators generate these as fast as the cpu will allow.

为此，是否有人有最好的/最快/最简单的方法来编写大量numpy数组来进行redis的构想或经验?这些都在同一台机器上，但是以后可以在一组云服务器上.代码示例欢迎您！

To this end, does anyone have good ideas or experience of the best/fastest/simplest way to write a lot of numpy arrays to redis. This is all on the same machine, but later, could be on a set of cloud servers. Code samples welcome!

推荐答案

我不知道它是否最快，但是您可以尝试类似的方法...

I don't know if it is fastest, but you could try something like this...

将Numpy数组存储到Redis就像这样-请参见函数toRedis():

Storing a Numpy array to Redis goes like this - see function toRedis():

• 获取Numpy数组的形状并进行编码
• 将Numpy数组作为字节追加到形状上
• 将编码后的数组存储在提供的键下

检索一个Numpy数组是这样的-请参见函数fromRedis():

Retrieving a Numpy array goes like this - see function fromRedis():

• 从Redis检索与提供的密钥相对应的编码字符串
• 从字符串中提取Numpy数组的形状
• 提取数据并重新填充Numpy数组，重塑为原始形状

#!/usr/bin/env python3

import struct
import redis
import numpy as np

def toRedis(r,a,n):
   """Store given Numpy array 'a' in Redis under key 'n'"""
   h, w = a.shape
   shape = struct.pack('>II',h,w)
   encoded = shape + a.tobytes()

   # Store encoded data in Redis
   r.set(n,encoded)
   return

def fromRedis(r,n):
   """Retrieve Numpy array from Redis key 'n'"""
   encoded = r.get(n)
   h, w = struct.unpack('>II',encoded[:8])
   a = np.frombuffer(encoded, dtype=np.uint16, offset=8).reshape(h,w)
   return a

# Create 80x80 numpy array to store
a0 = np.arange(6400,dtype=np.uint16).reshape(80,80) 

# Redis connection
r = redis.Redis(host='localhost', port=6379, db=0)

# Store array a0 in Redis under name 'a0array'
toRedis(r,a0,'a0array')

# Retrieve from Redis
a1 = fromRedis(r,'a0array')

np.testing.assert_array_equal(a0,a1)

通过将Numpy数组的dtype与形状一起编码，可以增加灵活性.我之所以没有这样做，是因为您可能已经知道所有数组都是一种特定的类型，然后代码会变得更大并且更无缘无故地难以阅读.

You could add more flexibility by encoding the dtype of the Numpy array along with the shape. I didn't do that because it may be the case that you already know all your arrays are of one specific type and then the code would just be bigger and harder to read for no reason.

现代iMac上的基准测试:

80x80 Numpy array of np.uint16   => 58 microseconds to write
200x200 Numpy array of np.uint16 => 88 microseconds to write

---

关键字:Python，Numpy，Redis，数组，序列化，序列化，键，incr，唯一

---

Keywords: Python, Numpy, Redis, array, serialise, serialize, key, incr, unique

这篇关于在Redis中存储Numpy数组的最快方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！