Stackexchange.Redis 中的流水线与批处理 [英] Pipelining vs Batching in Stackexchange.Redis

问题描述

我试图在尽可能短的时间内插入大量(-ish)元素，我尝试了以下两种选择:

I am trying to insert a large(-ish) number of elements in the shortest time possible and I tried these two alternatives:

1) 流水线:

List<Task> addTasks = new List<Task>();
for (int i = 0; i < table.Rows.Count; i++)
{
    DataRow row = table.Rows[i];
    Task<bool> addAsync = redisDB.SetAddAsync(string.Format(keyFormat, row.Field<int>("Id")), row.Field<int>("Value"));
    addTasks.Add(addAsync);
}
Task[] tasks = addTasks.ToArray();
Task.WaitAll(tasks);

2) 批处理:

List<Task> addTasks = new List<Task>();
IBatch batch = redisDB.CreateBatch();
for (int i = 0; i < table.Rows.Count; i++)
{
    DataRow row = table.Rows[i];
    Task<bool> addAsync = batch.SetAddAsync(string.Format(keyFormat, row.Field<int>("Id")), row.Field<int>("Value"));
    addTasks.Add(addAsync);
}
batch.Execute();
Task[] tasks = addTasks.ToArray();
Task.WaitAll(tasks);

我没有注意到任何显着的时间差异(实际上我希望批处理方法更快):对于大约 250K 的插入，我得到大约 7 秒的流水线和大约 8 秒的批处理.

I am not noticing any significant time difference (actually I expected the batch method to be faster): for approx 250K inserts I get approx 7 sec for pipelining vs approx 8 sec for batching.

阅读有关流水线的文档，

Reading from the documentation on pipelining,

"使用流水线允许我们将两个请求都发送到网络上立即消除大部分延迟.此外，它还有助于减少数据包碎片:单独发送 20 个请求(等待每个响应)将需要至少 20 个数据包，但 20在管道中发送的请求可以放入更少的数据包(也许甚至只有一个)."

"Using pipelining allows us to get both requests onto the network immediately, eliminating most of the latency. Additionally, it also helps reduce packet fragmentation: 20 requests sent individually (waiting for each response) will require at least 20 packets, but 20 requests sent in a pipeline could fit into much fewer packets (perhaps even just one)."

对我来说，这听起来很像批处理行为.我想知道这两者在幕后是否有很大的不同，因为在对 procmon 的简单检查中，我看到两个版本上的 TCP Send 数量几乎相同.>

To me, this sounds a lot like the a batching behaviour. I wonder if behind the scenes there's any big difference between the two because at a simple check with procmon I see almost the same number of TCP Sends on both versions.

推荐答案

在幕后，SE.Redis 做了相当多的工作来尝试避免数据包碎片，因此它在您的应用程序中非常相似也就不足为奇了案件.批处理和扁平流水线的主要区别是:

Behind the scenes, SE.Redis does quite a bit of work to try to avoid packet fragmentation, so it isn't surprising that it is quite similar in your case. The main difference between batching and flat pipelining are:

• 批处理永远不会与同一个多路复用器上的竞争操作交错(尽管它可能在服务器上交错；为了避免这种情况，您需要使用 multi/exec 交易或 Lua 脚本)
• 批处理将始终避免数据包过小的机会，因为它提前知道所有数据
• 但与此同时，在发送任何内容之前必须完成整个批次，因此这需要更多的内存缓冲并可能人为地引入延迟

• a batch will never be interleaved with competing operations on the same multiplexer (although it may be interleaved at the server; to avoid that you need to use a multi/exec transaction or a Lua script)
• a batch will be always avoid the chance of undersized packets, because it knows about all the data ahead of time
• but at the same time, the entire batch must be completed before anything can be sent, so this requires more in-memory buffering and may artificially introduce latency

在大多数情况下，避免批处理会更好，因为 SE.Redis 在添加工作时自动实现了它所做的大部分工作.

In most cases, you will do better by avoiding batching, since SE.Redis achieves most of what it does automatically when simply adding work.

作为最后一点；如果您想避免本地开销，最后一种方法可能是:

As a final note; if you want to avoid local overhead, one final approach might be:

redisDB.SetAdd(string.Format(keyFormat, row.Field<int>("Id")),
    row.Field<int>("Value"), flags: CommandFlags.FireAndForget);

这会将所有内容发送出去，既不等待响应也不分配不完整的 Task 来表示未来的值.您可能希望在结尾处执行Ping 之类的操作 一劳永逸，以检查服务器是否仍在与您通话.请注意，使用即发即弃确实意味着您不会注意到报告的任何服务器错误.

This sends everything down the wire, neither waiting for responses nor allocating incomplete Tasks to represent future values. You might want to do something like a Ping at the end without fire-and-forget, to check the server is still talking to you. Note that using fire-and-forget does mean that you won't notice any server errors that get reported.

这篇关于Stackexchange.Redis 中的流水线与批处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！