TakeLock 中的 protobuf-net 并发性能问题 [英] protobuf-net concurrent performance issue in TakeLock

问题描述

我们使用 protobuf-net 在服务之间发送日志消息.在分析压力测试时，在高并发下，我们看到 CPU 使用率非常高，而 RuntimeTypeModel 中的 TakeLock 是罪魁祸首.热调用堆栈看起来像:

We're using protobuf-net for sending log messages between services. When profiling stress testing, under high concurrency, we see very high CPU usage and that TakeLock in RuntimeTypeModel is the culprit. The hot call stack looks something like:

*Our code...*
ProtoBuf.Serializer.SerializeWithLengthPrefix(class System.IO.Stream,!!0,valuetype ProtoBuf.PrefixStyle)
ProtoBuf.Serializer.SerializeWithLengthPrefix(class System.IO.Stream,!!0,valuetype ProtoBuf.PrefixStyle,int32)
ProtoBuf.Meta.TypeModel.SerializeWithLengthPrefix(class System.IO.Stream,object,class System.Type,valuetype ProtoBuf.PrefixStyle,int32)
ProtoBuf.Meta.TypeModel.SerializeWithLengthPrefix(class System.IO.Stream,object,class System.Type,valuetype ProtoBuf.PrefixStyle,int32,class ProtoBuf.SerializationContext)
ProtoBuf.ProtoWriter.WriteObject(object,int32,class ProtoBuf.ProtoWriter,valuetype  ProtoBuf.PrefixStyle,int32)
ProtoBuf.BclHelpers.WriteNetObject(object,class ProtoBuf.ProtoWriter,int32,valuetype 
ProtoBuf.BclHelpers/NetObjectOptions)
ProtoBuf.Meta.TypeModel.GetKey(class System.Type&)
ProtoBuf.Meta.RuntimeTypeModel.GetKey(class System.Type,bool,bool)
ProtoBuf.Meta.RuntimeTypeModel.FindOrAddAuto(class System.Type,bool,bool,bool)
ProtoBuf.Meta.RuntimeTypeModel.TakeLock(int32&)
[clr.dll]

我看到我们可以使用新的预编译器来提高速度，但我想知道这是否会解决这个问题(听起来它不使用反射)；整合这个对我来说会有点工作，所以我还没有测试过.我还看到了调用 Serializer.PrepareSerializer 的选项.我最初的(小规模)测试并没有让准备工作看起来很有希望.

I see that we can use the new precompiler to get a speed boost, but I'm wondering if that will get rid of the issue (sounds like it doesn't use reflection); it would be a bit of work for me to integrate this, so I haven't tested it yet. I also see the option to call Serializer.PrepareSerializer. My initial (small scale) testing didn't make the prepare seem promising.

关于我们正在序列化的类型的更多信息:

A little more info about the type we're serializing:

[ProtoContract]
public class SomeMessage
{
    [ProtoMember(1)]
    public SomeEnumType SomeEnum { get; set; }

    [ProtoMember(2)]
    public long SomeId{ get; set; }

    [ProtoMember(3)]
    public string SomeString{ get; set; }

    [ProtoMember(4)]
    public DateTime SomeDate { get; set; }

    [ProtoMember(5, DynamicType = true, OverwriteList = true)]
    public Collection<object> SomeArguments
}

感谢您的帮助！

更新 9/17

感谢您的回复！我们将尝试您建议的解决方法，看看是否能解决问题.

Thanks for your response! We're going to try the workaround you suggest and see if that fixes things.

这段代码存在于我们的日志系统中，所以在 SomeMessage 示例中，SomeString 实际上是一个格式字符串(例如Hello {0}")，而 SomeArguments 集合是用于填充格式字符串的对象列表，只是像String.Format.在序列化之前，我们查看每个参数并调用DynamicSerializer.IsKnownType(argument.GetType())，如果不知道，我们先将其转换为字符串.我没有看过数据的比率，但我很确定我们有很多不同的字符串作为参数传入.

This code lives in our logging system so, in the SomeMessage example, SomeString is really a format string (e.g. "Hello {0}") and the SomeArguments collection is a list of objects used to fill in the format string, just like String.Format. Before we serialize, we look at each argument and call DynamicSerializer.IsKnownType(argument.GetType()), if it isn't known, we convert it to a string first. I haven't looked at the ratios of data, but I'm pretty sure we have a lot of different strings coming in as arguments.

让我知道这是否有帮助.如果您需要，我会尝试获取更多详细信息.

Let me know if this helps. If you need, I'll try to get more details.

推荐答案

TakeLock 仅在改变模型时使用，例如因为它正在看到一个类型首次.在第一次使用特定类型后，您通常不会看到 TakeLock.在大多数情况下，使用 Serializaer.PrepareSerializer() 应该执行所有必要的初始化(对于您正在使用的任何其他合约也类似).

TakeLock is only used when it is changing the model, for example because it is seeing a type for the first time. You shouldn't normally see TakeLock after the first time a particular type has been used. In most cases, using Serializaer.PrepareSerializer<SomeMessage>() should perform all the necessary initialization (and similar for any other contracts you are using).

但是！我想知道这是否也与您使用DynamicType有关；这里使用的实际对象是什么?可能是我需要调整这里的逻辑，这样它就不会在这一步上花费任何时间.如果您让我知道实际对象(以便我可以重现)，我将尝试运行一些测试.

However! I wonder if perhaps this is also related to your use of DynamicType; what are the actual objects being used here? It might be that I need to tweak the logic here, so that it doesn't spend any time on that step. If you let me know the actual objects (so I can repro), I will try to run some tests.

至于预编译器是否会改变这个；是的.一个完全编译的静态模型有一个完全不同的ProtoBuf.Meta.TypeModel.GetKey方法实现，所以它永远不会调用TakeLock(你不需要保护永远不会改变的模型！).但是你实际上可以做一些非常相似的事情， 不需要使用预编译.考虑以下情况，作为应用初始化的一部分运行:

As for whether the precompiler would change this; yes it would. A fully compiled static model has a completely different implementation of the ProtoBuf.Meta.TypeModel.GetKey method, so it would never call TakeLock (you don't need to protect a model that can never change!). But you can actuallydo something very similar without needing to use precompile. Consider the following, run as part of your app's initialization:

static readonly TypeModel serializer;
...
var model = TypeModel.Create();
model.Add(typeof(SomeMessage), true);
// TODO add other contracts you use here
serializer = model.Compile();

这将在内存中创建一个完全静态编译的序列化程序程序集(而不是一个编译了单个操作的可变模型).如果您现在使用 serializer.Serialize(...) 而不是 Serializer.Serialize(即存储的 TypeModel 上的实例方法，而不是Serializer 上的静态方法)然后它本质上会做一些与预编译器"非常相似的事情，但不需要实际预编译它(显然这只能在完整".NET).这将永远不会调用 TakeLock，因为它运行的是固定模型，而不是灵活模型.但是，它确实要求您知道您使用的合同类型.您可以使用反射来查找这些，通过查找具有给定属性的所有类型:

This will create a fully static-compiled serializer assembly in memory (instead of a mutable model with individual operations compiled). If you now use serializer.Serialize(...) instead of Serializer.Serialize (i.e. the instance method on your stored TypeModel rather than the static method on Serializer) then it will essentially be doing something very similar to "precompiler", but without the need to actualy precompile it (obviously this will only be available on "full" .NET). This will then never call TakeLock, as it is running a fixed model, rather than a flexible model. It does, however, require you to know what contract-types you use. You could use reflection to find these, by looking for all those types with a given attribute:

static readonly TypeModel serializer;
...
var model = TypeModel.Create();
Type attributeType = typeof(ProtoContractAttribute);
foreach (var type in typeof(SomeMessage).Assembly.GetTypes()) {
    if (Attribute.IsDefined(type, attributeType)) {
        model.Add(type, true);
    }
}
serializer = model.Compile();

但强调:以上是一种解决方法；听起来好像有一个小故障，如果我能看到一个实际发生的例子，我会很高兴地进行调查；最重要的是:SomeArguments 中的对象是什么?

But emphasis: the above is a workaround; it sounds like there's a glitch, which I'll happily investigate if I can see an example where it actually happens; most importantly: what are the objects in SomeArguments?

这篇关于TakeLock 中的 protobuf-net 并发性能问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！