将字典中的数据批量设置到Redis中 [英] Batch set data from Dictionary into Redis

问题描述

我正在使用 StackExchange Redis DB 使用 Batch 插入键值对字典，如下所示:

I am using StackExchange Redis DB to insert a dictionary of Key value pairs using Batch as below:

private static StackExchange.Redis.IDatabase _database;
public void SetAll<T>(Dictionary<string, T> data, int cacheTime)
{
    lock (_database)
    {
        TimeSpan expiration = new TimeSpan(0, cacheTime, 0);
        var list = new List<Task<bool>>();
        var batch = _database.CreateBatch();               
        foreach (var item in data)
        {
            string serializedObject = JsonConvert.SerializeObject(item.Value, Formatting.Indented,
        new JsonSerializerSettings { ContractResolver = new SerializeAllContractResolver(), ReferenceLoopHandling = ReferenceLoopHandling.Ignore });

            var task = batch.StringSetAsync(item.Key, serializedObject, expiration);
            list.Add(task);
            serializedObject = null;
        }
        batch.Execute();

        Task.WhenAll(list.ToArray());
    }
}

我的问题:设置字典的 350 项大约需要 7 秒.

My problem: It takes around 7 seconds to set just 350 items of dictionary.

我的问题:这是将批量项目设置到 Redis 的正确方法还是有更快的方法?任何帮助表示赞赏.谢谢.

My question: Is this the right way to set bulk items into Redis or is there a quicker way to do this? Any help is appreciated. Thanks.

推荐答案

只是"是一个非常相对的术语，如果没有更多的上下文，就没有真正的意义，特别是:这些有效载荷有多大?

"just" is a very relative term, and doesn't really make sense without more context, in particular: how big are these payloads?

但是，澄清几点以帮助您调查:

however, to clarify a few points to help you investigate:

• 没有必要锁定IDatabase，除非这纯粹是为了您自己的目的；SE.Redis 在内部处理线程安全，旨在供竞争线程使用
• 目前，您的时间安排将包括所有序列化代码 (JsonConvert.SerializeObject)；这会加起来，特别是如果您的对象很大；为了得到一个不错的衡量标准，我强烈建议您分别
安排序列化和 redis 的时间
• batch.Execute() 方法使用管道 API 并且不等待调用之间的响应，因此:您看到的时间不是累积效应的延迟；只剩下本地 CPU(用于序列化)、网络带宽和服务器 CPU；客户端库工具不会影响任何这些事情
• 有一个 StringSet 重载，它接受 KeyValuePair[];你可以选择使用它而不是批处理，但这里唯一的区别是它是可变的MSET而不是多个SET；无论哪种方式，您都会在持续时间内阻止其他调用者的连接(因为批处理的目的是使命令连续)
• 您实际上不需要在这里使用 CreateBatch，尤其，因为您正在锁定数据库(但我仍然建议您不要不需要这样做)；CreateBatch 的目的是制作一系列命令 sequential，但我不认为你在这里需要它；您可以依次为每个命令使用 _database.StringSetAsync，这将 也 具有以下优势:您可以并行地运行序列化正在发送的上一个命令 - 除了删除 CreateBatch 调用之外，它允许您重叠序列化(CPU 绑定)和 redis 操作(IO 绑定)而无需任何工作；这也意味着您不会独占来自其他呼叫者的连接

• there is no need to lock an IDatabase unless that is purely for your own purposes; SE.Redis deals with thread safety internally and is intended to be used by competing threads
• at the moment, your timing of this will include all the serialization code (JsonConvert.SerializeObject); this will add up, especially if your objects are big; to get a decent measure, I strongly suggest you time the serialization and redis times separately
• the batch.Execute() method uses a pipeline API and does not wait for responses between calls, so: the time you're seeing is not the cumulative effect of latency; that leaves just local CPU (for serialization), network bandwidth, and server CPU; the client library tools can't impact any of those things
• there is a StringSet overload that accepts a KeyValuePair<RedisKey, RedisValue>[]; you could choose to use this instead of a batch, but the only difference here is that it is the varadic MSET rather than muliple SET; either way, you'll be blocking the connection for other callers for the duration (since the purpose of batch is to make the commands contiguous)
• you don't actually need to use CreateBatch here, especially since you're locking the database (but I still suggest you don't need to do this); the purpose of CreateBatch is to make a sequence of commands sequential, but I don't see that you need this here; you could just use _database.StringSetAsync for each command in turn, which would also have the advantage that you'd be running serialization in parallel to the previous command being sent - it would allow you to overlap serialization (CPU bound) and redis ops (IO bound) without any work except to delete the CreateBatch call; this will also mean that you don't monopolize the connection from other callers

所以；我要做的第一件事是删除一些代码:

So; the first thing I would do would be to remove some code:

private static StackExchange.Redis.IDatabase _database;
static JsonSerializerSettings _redisJsonSettings = new JsonSerializerSettings {
    ContractResolver = new SerializeAllContractResolver(),
    ReferenceLoopHandling = ReferenceLoopHandling.Ignore };

public void SetAll<T>(Dictionary<string, T> data, int cacheTime)
{
    TimeSpan expiration = new TimeSpan(0, cacheTime, 0);
    var list = new List<Task<bool>>();
    foreach (var item in data)
    {
        string serializedObject = JsonConvert.SerializeObject(
            item.Value, Formatting.Indented, _redisJsonSettings);

        list.Add(_database.StringSetAsync(item.Key, serializedObject, expiration));
    }
    Task.WhenAll(list.ToArray());
}

我要做的第二件事是将序列化时间与 redis 工作分开进行.

The second thing I would do would be to time the serialization separately to the redis work.

我要做的第三件事是看看我是否可以序列化为 MemoryStream，理想情况下，我可以重复使用它 - 避免 string 分配和 UTF-8 编码:

The thrid thing I would do would be to see if I can serialize to a MemoryStream instead, ideally one that I can re-use - to avoid the string alocation and UTF-8 encode:

using(var ms = new MemoryStream())
{
    foreach (var item in data)
    {
        ms.Position = 0;
        ms.SetLength(0); // erase existing data
        JsonConvert.SerializeObject(ms,
            item.Value, Formatting.Indented, _redisJsonSettings);

        list.Add(_database.StringSetAsync(item.Key, ms.ToArray(), expiration));
    }
}

这篇关于将字典中的数据批量设置到Redis中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！