在C#中读取事件中心存档文件 [英] Reading Event Hub Archive File in C#

问题描述

C#中是否有用于读取Azure Event Hub存档文件(Avro格式)的示例代码?

Is there any sample code in C# for reading the Azure Event Hub Archive files (Avro format)?

我正在尝试使用Microsoft.Hadoop.Avro库.我使用产生此结果的java avro工具转储了该模式:

I am trying to use the Microsoft.Hadoop.Avro library. I dumped the schema out using a java avro tool which produces this:

{

                ""type"":""record"",
                ""name"":""EventData"",
                ""namespace"":""Microsoft.ServiceBus.Messaging"",
                ""fields"":[
                             {""name"":""SequenceNumber"",""type"":""long""},
                             {""name"":""Offset"",""type"":""string""},
                             {""name"":""EnqueuedTimeUtc"",""type"":""string""},
                             {""name"":""SystemProperties"",""type"":{ ""type"":""map"",""values"":[""long"",""double"",""string"",""bytes""]}},
                             {""name"":""Properties"",""type"":{ ""type"":""map"",""values"":[""long"",""double"",""string"",""bytes"", ""null""]}},
                             {""name"":""Body"",""type"":[""null"",""bytes""]}
                         ]
                }

但是，当尝试反序列化文件以像这样读回数据时:

However, when trying to deserialize the file to read the data back in like this:

using (var reader = AvroContainer.CreateReader<EventData>(stream))
            {
                using (var streamReader = new SequentialReader<EventData>(reader))
                {
                    foreach (EventData dta in streamReader.Objects)
                    {
                        //stuff here
                    }

                }
            }

当传递生产者端使用的实际EventData类型时，它不起作用，因此我尝试创建一个标记有DataContract属性的特殊类，如下所示:

It doesn't work when passing the actual EventData type used on the Producer side so I tried to create a special class marked up with DataContract attributes like this:

[DataContract(Namespace = "Microsoft.ServiceBus.Messaging")]
public class EventData
{
    [DataMember(Name = "SequenceNumber")]
    public long SequenceNumber { get; set; }

    [DataMember(Name = "Offset")]
    public string Offset { get; set; }

    [DataMember(Name = "EnqueuedTimeUtc")]
    public string EnqueuedTimeUtc { get; set; }

    [DataMember(Name = "Body")]
    public ArraySegment<byte> Body { get; set; }

    //[DataMember(Name = "SystemProperties")]
    //public SystemPropertiesCollection SystemProperties { get; set; }

    //[DataMember(Name = "Properties")]
    //public IDictionary<string, object> Properties { get; set; }
}

出现以下错误:

System.Runtime.Serialization.SerializationException occurred
Message=Cannot match the union schema.

对于这种使用C#读取Avro存档文件的用例，是否有理由没有MS提供示例代码?

Is there a reason no sample code exists from MS for this use case of reading the Avro archive files using C#?

推荐答案

如果您尝试使用Microsoft.Hadoop.Avro库读取Avro文件，则可以使用以下类:

If you're trying to read the Avro files using Microsoft.Hadoop.Avro library, you can use the following class:

[DataContract(Name = "EventData", Namespace = "Microsoft.ServiceBus.Messaging")]
class EventData
{
    [DataMember(Name = "SequenceNumber")]
    public long SequenceNumber { get; set; }

    [DataMember(Name = "Offset")]
    public string Offset { get; set; }

    [DataMember(Name = "EnqueuedTimeUtc")]
    public DateTime EnqueuedTimeUtc { get; set; }

    [DataMember(Name = "SystemProperties")]
    public Dictionary<string, object> SystemProperties { get; set; }

    [DataMember(Name = "Properties")]
    public Dictionary<string, object> Properties { get; set; } 

    [DataMember(Name = "Body")]
    public byte[] Body { get; set; }

    public EventData(dynamic record)
    {
        SequenceNumber = (long)record.SequenceNumber;
        Offset = (string)record.Offset;
        DateTime.TryParse((string)record.EnqueuedTimeUtc, out var enqueuedTimeUtc);
        EnqueuedTimeUtc = enqueuedTimeUtc;
        SystemProperties = (Dictionary<string, object>)record.SystemProperties;
        Properties = (Dictionary<string, object>)record.Properties;
        Body = (byte[])record.Body;
    }

}

读取avro文件时，可以将其读取为动态对象，然后对其进行序列化.这是一个示例:

When you're reading your avro file, you can read it as a dynamic object and then serialize it. Here's an example:

var reader = AvroContainer.CreateGenericReader(stream);
while (reader.MoveNext()) 
{
   foreach (dynamic record in reader.Current.Objects)
   {
       var eventData = new EventData(record);
       var sequenceNumber = eventData.SequenceNumber;
       var bodyText = Encoding.UTF8.GetString(eventData.Body);
       var properties = eventData.Properties;
       var sysProperties = eventData.SystemProperties;
   }
}

您可以参考此答案以了解更多详细信息.

You can refer to this answer for more details.

这篇关于在C#中读取事件中心存档文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！