部分和异步反序列化的C＃与protobuf网 [英] Partial and asynchronous deserialization c# with protobuf-net

问题描述

我有具有以下结构的文件：

I'm have a file with the following structure:

[ProtoContract]
public class Data
{
    [ProtoMember(1)]
    public string Header { get; set; }

    [ProtoMember(2)]
    public byte[] Body { get; set; }
}

读取/将数据写入到文件中的代码在运行asp.net的MVC的WebAPI上下文。我试图让每一个阻塞IO异步减少阻塞，获得最佳的可扩展性。读取和写入的文件不支持ReadAsync，WriteAsync和CopyToAsync。

The code that reads / writes the data to a file is running on a asp.net mvc webapi context. I'm trying to keep every blocking IO async to minimize blocking and achieve the best scalability. Reading and writing from files does support ReadAsync, WriteAsync and CopyToAsync.

主体可以是相当大（>>头），我只需要读取人体如果标题匹配一些具体的标准。

The body can be reasonably large (>> header) and I only need to read the body if the header matches some specific criteria.

我能够部分地理解，同步反序列化头和读取，并通过使用该方法反序列化体以相同的方式在反序列化的二进制文件的一部分

I can partially read and deserialize the header synchronously and read and deserialize the body the same way by using the approach explained in Deserialize part of a binary file

我如何使用异步文件IO做完全一样的，阅读和反序列化头部异步和读取和反序列化的身体以同样的方式？

How can I use asynchronous file IO to do exactly the same, reading and deserializing the header Async and reading and deserializing the body the same way?

我读过异步串行化的protobuf 是不是一种选择。

I've read Asynchronous protobuf serialization is not an option.

推荐答案

从技术上protobuf的字段可以是外的秩序，但在大多数情况下（包括您展示一个），我们可以合理假设字段按顺序（让他们的唯一出路序这里将单独序列两个半班和串联的结果，这是在protobuf的规范技术上有效）。

Technically protobuf fields can be out-of-order, but in most cases (including the one you show) we can reasonable assume the fields are in-order (the only way to get them out-of-order here would be to separately serialize two half-classes and concatenate the results, which is technically valid in the protobuf specification).

左右;我们将拥有的是：

So; what we will have is:

• 一个varint表示：字段1，串 - 总是十进制 10


• 一个varint表示是，标题的长度


• 一字节，UTF-8编码的标题


• 一个varint表示：场2，串 - 总是十进制 18


• 一个varint表示b 身体

的长度
• b的字节，体

• a varint denoting: field 1, string - always decimal 10
• a varint denoting "a", the length of the header
• "a" bytes, the UTF-8 encoded header
• a varint denoting: field 2, string - always decimal 18
• a varint denoting "b", the length of the body
• "b" bytes, the body

我们可以大概认为a是> = 0 和< int.MaxValue - 这意味着将需要最多5个字节编码;所以，如果你缓冲的至少的6个字节，你将有足够的信息来了解头部有多大。当然，这也从技术上可能包含身体的一部分，所以你需要保持保持吧！但是，如果你有一个同步过异步流，你可以阅读的只是的通过像流的一部分：

We can probably assume that "a" is >= 0 and < int.MaxValue - which means it will take at most 5 bytes to encode; so, if you buffer at least 6 bytes, you will have enough information to know how large the header is. Of course, it could technically also contain part of the body, so you'd need to keep hold of it! But if you had a sync-over-async Stream, you can read just that part of the stream by something like:

int protoHeader = ProtoReader.DirectReadVarintInt32(stream); // 10
int headerLength = ProtoReader.DirectReadVarintInt32(stream);
string header = ProtoReader.DirectReadString(stream, headerLength);

或者，如果同步在异步是棘手的，明确阅读：

Or if the "sync over async" is tricky, explicit reading:

static byte[] ReadAtLeast6()
{
    return new byte[] { 0x0A, 0x0B, 0x68, 0x65, 0x6C, 0x6C, 0x6F };
}
static byte[] ReadMore(int bytes)
{
    return new byte[] { 0x20, 0x77, 0x6F, 0x72, 0x6C, 0x64 };
}
static void Main()
{
    // pretend we read 7 bytes async
    var data = ReadAtLeast6();
    using (var ms = new MemoryStream())
    {
        ms.Write(data, 0, data.Length);
        ms.Position = 0;
        int protoHeader = ProtoReader.DirectReadVarintInt32(ms); // 10
        int headerLength = ProtoReader.DirectReadVarintInt32(ms); // 11

        int needed = (headerLength + (int)ms.Position) - data.Length; // 6 more
        var pos = ms.Position;
        ms.Seek(0, SeekOrigin.End);
        data = ReadMore(needed);
        ms.Write(data, 0, needed);
        ms.Position = pos;
        string header = ProtoReader.DirectReadString(ms, headerLength);
    }
}

这篇关于部分和异步反序列化的C＃与protobuf网的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！