实时股票应用的Redis密钥设计 [英] Redis key design for real-time stock application

问题描述

我正在尝试构建一个实时股票应用程序.每秒我可以从网络服务中获取一些数据，如下所示:

I am trying to build a real-time stock application. Every seconds I can get some data from web service like below:

[{"amount":"20","date":1386832664,"price":"183.8","tid":5354831,"type":"sell"},{"amount":"22","date":1386832664,"price":"183.61","tid":5354833,"type":"buy"}]

[{"amount":"20","date":1386832664,"price":"183.8","tid":5354831,"type":"sell"},{"amount":"22","date":1386832664,"price":"183.61","tid":5354833,"type":"buy"}]

tid 是股票买卖的票号；日期是从 1970.1.1 开始的第二个；价格/数量是指交易的股票价格和数量.

tid is the ticket ID for stock buying and selling; date is the second from 1970.1.1; price/amount is at what price and how many stock traded.

需求

我的要求是每分钟/5分钟/小时/天实时显示用户最高/最低价格；实时向用户显示每分钟/5分钟/小时/天的金额总和.

My requirement is show user highest/lowest price at every minute/5 minutes/hour/day in real-time; show user the sum of amount in every minute/5 minutes/hour/day in real-time.

问题

我的问题是如何将数据存储到redis，以便我可以轻松快速地从DB获取不同时期的最高/最低交易.

My question is how to store the data to redis, so that I can easily and quickly get highest/lowest trade from DB for different periods.

我的设计如下:

[date]:[tid]:amount
[date]:[tid]:price
[date]:[tid]:type

我是 redis 新手.如果设计是这样，那意味着我需要使用排序集，会不会有任何性能问题?或者有没有其他方法可以获得不同时期的最高/最低价格.

I am new in redis. If the design is this is that means I need to use sorted set, will there any performance issue? Or is there any other way to get highest/lowest price for different periods.

期待您的建议和设计.

推荐答案

我的建议是存储您感兴趣的所有间隔的 min/max/total，并针对每个到达的数据点更新当前间隔.为了避免读取之前数据进行比较时的网络延迟，您可以使用 Lua 脚本完全在 Redis 服务器内完成.

My suggestion is to store min/max/total for all intervals you are interested in and update it for current ones with every arriving data point. To avoid network latency when reading previous data for comparison, you can do it entirely inside Redis server using Lua scripting.

每个数据点(或者更糟糕的是，每个数据点字段)一个键会消耗太多内存.为了获得最佳结果，您应该将其分组为小列表/哈希(请参阅 http://redis.io/topics/内存优化).Redis 只允许在其数据结构中进行一层嵌套:如果您的数据有多个字段，并且您想为每个键存储多个项目，则需要自己以某种方式对其进行编码.幸运的是，标准的 Redis Lua 环境包括 msgpack 支持，这是一种非常高效的二进制 JSON 式格式.您的示例中使用 msgpack 按原样"编码的 JSON 条目的长度为 52-53 个字节.我建议按时间分组，以便每个键有 100-1000 个条目.假设一分钟间隔符合此要求.那么密钥方案将是这样的:

One key per data point (or, even worse, per data point field) is going to consume too much memory. For the best results, you should group it into small lists/hashes (see http://redis.io/topics/memory-optimization). Redis only allows one level of nesting in its data structures: if you data has multiple fields and you want to store more than one item per key, you need to somehow encode it yourself. Fortunately, standard Redis Lua environment includes msgpack support which is very a efficient binary JSON-like format. JSON entries in your example encoded with msgpack "as is" will be 52-53 bytes long. I suggest grouping by time so that you have 100-1000 entries per key. Suppose one-minute interval fits this requirement. Then the keying scheme would be like this:

YYmmddHHMMSS — 从 tid 到给定分钟的 msgpack 编码数据点的哈希值.5m:YYmmddHHMM, 1h:YYmmddHH, 1d:YYmmdd — 包含 min, 的窗口数据哈希max, sum 字段.

YYmmddHHMMSS — a hash from tid to msgpack-encoded data points for the given minute. 5m:YYmmddHHMM, 1h:YYmmddHH, 1d:YYmmdd — window data hashes which contain min, max, sum fields.

让我们看一个示例 Lua 脚本，它将接受一个数据点并根据需要更新所有键.由于 Redis 脚本的工作方式，我们需要明确传递脚本将访问的所有键的名称，即实时数据和所有三个窗口键.Redis Lua 也有可用的 JSON 解析库，所以为了简单起见，我们假设我们只传递 JSON 字典.这意味着我们必须对数据进行两次解析:在应用程序端和Redis 端，但其性能影响尚不清楚.

Let's look at a sample Lua script that will accept one data point and update all keys as necessary. Due to the way Redis scripting works we need to explicitly pass the names of all keys that will be accessed by the script, i.e. the live data and all three window keys. Redis Lua has also JSON parsing library available, so for the sake of simplicity let's assume we just pass it JSON dictionary. That means that we have to parse data twice: on the application side and on the Redis side, but the performance effects of it are not clear.

local function update_window(winkey, price, amount)
    local windata = redis.call('HGETALL', winkey)
    if price > tonumber(windata.max or 0) then
        redis.call('HSET', winkey, 'max', price)
    end
    if price < tonumber(windata.min or 1e12) then
        redis.call('HSET', winkey, 'min', price)
    end
    redis.call('HSET', winkey, 'sum', (windata.sum or 0) + amount)
end

local currkey, fiveminkey, hourkey, daykey = unpack(KEYS)
local data = cjson.decode(ARGV[1])
local packed = cmsgpack.pack(data)
local tid = data.tid
redis.call('HSET', currkey, tid, packed)
local price = tonumber(data.price)
local amount = tonumber(data.amount)
update_window(fiveminkey, price, amount)
update_window(hourkey, price, amount)
update_window(daykey, price, amount)

这种设置每秒可以进行数千次更新，内存不是很大，并且可以立即检索窗口数据.

This setup can do thousands of updates per second, not very hungry on memory, and window data can be retrieved instantly.

更新:在内存部分，如果你想存储更多几百万，每点 50-60 个字节仍然很多.对于这种数据，我认为您可以使用自定义二进制格式、增量编码以及随后使用诸如 snappy.这取决于您的要求，是否值得这样做.

UPDATE: On the memory part, 50-60 bytes per point is still a lot if you want to store more a few millions. With this kind of data I think you can get as low as 2-3 bytes per point using custom binary format, delta encoding, and subsequent compression of chunks using something like snappy. It depends on your requirements, whether it's worth doing this.

这篇关于实时股票应用的Redis密钥设计的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！