Stackexchange.Redis中的流水线与批处理 [英] Pipelining vs Batching in Stackexchange.Redis

问题描述

我正在尝试在尽可能短的时间内插入大量(-ish)元素，并且尝试了以下两种选择:

I am trying to insert a large(-ish) number of elements in the shortest time possible and I tried these two alternatives:

1)流水线:

List<Task> addTasks = new List<Task>();
for (int i = 0; i < table.Rows.Count; i++)
{
    DataRow row = table.Rows[i];
    Task<bool> addAsync = redisDB.SetAddAsync(string.Format(keyFormat, row.Field<int>("Id")), row.Field<int>("Value"));
    addTasks.Add(addAsync);
}
Task[] tasks = addTasks.ToArray();
Task.WaitAll(tasks);

2)批处理:

List<Task> addTasks = new List<Task>();
IBatch batch = redisDB.CreateBatch();
for (int i = 0; i < table.Rows.Count; i++)
{
    DataRow row = table.Rows[i];
    Task<bool> addAsync = batch.SetAddAsync(string.Format(keyFormat, row.Field<int>("Id")), row.Field<int>("Value"));
    addTasks.Add(addAsync);
}
batch.Execute();
Task[] tasks = addTasks.ToArray();
Task.WaitAll(tasks);

我没有注意到任何明显的时差(实际上我希望批处理方法更快):对于大约250K的插入，流水处理大约需要7秒，而批处理大约需要8秒.

I am not noticing any significant time difference (actually I expected the batch method to be faster): for approx 250K inserts I get approx 7 sec for pipelining vs approx 8 sec for batching.

阅读有关流水线的文档，

Reading from the documentation on pipelining,

使用流水线使我们能够将两个请求都发送到网络上 立即消除大部分延迟.另外，它也 帮助减少数据包碎片:单独发送20个请求 (等待每个响应)至少需要20个数据包，但20个数据包 管道中发送的请求可以容纳更少的数据包(也许 甚至只有一个)."

"Using pipelining allows us to get both requests onto the network immediately, eliminating most of the latency. Additionally, it also helps reduce packet fragmentation: 20 requests sent individually (waiting for each response) will require at least 20 packets, but 20 requests sent in a pipeline could fit into much fewer packets (perhaps even just one)."

对我来说，这听起来很像批处理行为.我想知道两者之间是否有很大的不同，因为在简单检查procmon时，我发现两个版本的TCP Send几乎相同.

To me, this sounds a lot like the a batching behaviour. I wonder if behind the scenes there's any big difference between the two because at a simple check with procmon I see almost the same number of TCP Sends on both versions.

推荐答案

在后台，SE.Redis做了很多工作来避免数据包碎片化，因此它在您的应用中非常相似也就不足为奇了案子.批处理和平面流水处理之间的主要区别是:

Behind the scenes, SE.Redis does quite a bit of work to try to avoid packet fragmentation, so it isn't surprising that it is quite similar in your case. The main difference between batching and flat pipelining are:

• 批处理永远不会与同一多路复用器上的竞争操作交错(尽管它可以在服务器上交错；为避免您需要使用multi/exec事务或Lua脚本)
• 批次将始终避免出现数据包过小的机会，因为它可以提前知道所有数据
• 但是同时，必须先完成整个批处理，然后才能发送任何内容，因此这需要更多的内存缓冲，并且可能人为地引入延迟

• a batch will never be interleaved with competing operations on the same multiplexer (although it may be interleaved at the server; to avoid that you need to use a multi/exec transaction or a Lua script)
• a batch will be always avoid the chance of undersized packets, because it knows about all the data ahead of time
• but at the same time, the entire batch must be completed before anything can be sent, so this requires more in-memory buffering and may artificially introduce latency

在大多数情况下，您可以避免批处理，因为SE.Redis只需添加工作即可自动完成大部分工作，因此可以避免批处理.

In most cases, you will do better by avoiding batching, since SE.Redis achieves most of what it does automatically when simply adding work.

作为最后的提示；如果您想避免本地开销，那么最后一种方法可能是:

As a final note; if you want to avoid local overhead, one final approach might be:

redisDB.SetAdd(string.Format(keyFormat, row.Field<int>("Id")),
    row.Field<int>("Value"), flags: CommandFlags.FireAndForget);

这将一切发送出去，既不等待响应也不分配不完整的Task来表示将来的值.您可能想要在末尾执行Ping之类的操作，而无需忘记即插即用，以检查服务器是否仍在与您通信.请注意，使用即弃操作确实意味着您不会注意到任何报告的服务器错误.

This sends everything down the wire, neither waiting for responses nor allocating incomplete Tasks to represent future values. You might want to do something like a Ping at the end without fire-and-forget, to check the server is still talking to you. Note that using fire-and-forget does mean that you won't notice any server errors that get reported.

这篇关于Stackexchange.Redis中的流水线与批处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！