google protobuf 最大大小 [英] google protobuf maximum size
问题描述
我的 protobuf 消息中有一些重复的元素.在运行时，消息的长度可以是任何东西——我看到一些已经问过这样的问题——[1]:最大序列化 Protobuf 消息大小
I have some repeating elements in my protobuf message. At runtime the length of the message could be anything - I see some questions already asked like this one - [1]: Maximum serialized Protobuf message size
- 我有一个稍微不同的问题.如果我的 JMS(Java 消息传递服务)提供程序(在本例中是我的 weblogic 或 tibco jms 服务器)没有任何大小限制最大消息大小，将协议缓冲区编译器抱怨最大消息大小?
- 是否编码/解码的性能在大尺寸下严重受损(大约 10MB)...?
推荐答案
10MB 正在推动它，但你可能会没事的.
10MB is pushing it but you'll probably be OK.
Protobuf 有 2GB 的硬限制，因为许多实现使用 32 位有符号算法.出于安全原因，许多实现(尤其是 Google 提供的实现)默认将大小限制设为 64MB，不过您可以根据需要手动增加此限制.
Protobuf has a hard limit of 2GB, because many implementations use 32-bit signed arithmetic. For security reasons, many implementations (especially the Google-provided ones) impose a size limit of 64MB by default, although you can increase this limit manually if you need to.
实现本身不会减慢"大消息的速度，但问题是您必须始终一次解析整个消息，然后才能开始使用任何内容.这意味着整个消息必须适合 RAM(请记住，解析内存中的消息对象比原始序列化消息大得多)，即使您只关心您拥有的一个字段等待整个事情解析.
The implementation will not "slow down" with large messages per se, but the problem is that you must always parse an entire message at once before you can start using any of the content. This means the entire message must fit into RAM (keeping in mind that after parsing the in-memory message objects are much larger than the original serialized message), and even if you only care about one field you have to wait for the whole thing to parse.
一般来说，我建议根据经验将自己限制在 1MB 以内.除此之外，考虑将消息分成可以独立解析的多个块.然而，每个应用程序——对某些人来说，10MB 没什么大不了的，对其他人来说，1MB 已经太大了.您必须对自己的应用进行概要分析才能找到答案.
Generally I recommend trying to limit yourself to 1MB as a rule of thumb. Beyond that, think about splitting the message up into multiple chunks that can be parsed independently. However, every application -- for some, 10MB is no big deal, for others 1MB is already way too large. You'll have to profile your own app to find out.
我实际上看到过人们很乐意发送大于 1GB 的消息的情况，所以……它有效".
I've actually seen cases where people were happy sending messages larger than 1GB, so... it "works".
顺便提一下，Cap'n Proto 的设计与 Protobuf 非常相似，但最多可以支持 2^64 字节(每个 4GB 的 2^32 段)，它实际上允许您从消息中读取一个字段而无需解析整个消息(如果它在磁盘上的文件中，请使用 mmap()以避免阅读整个内容).
On a side note, Cap'n Proto has a very similar design to Protobuf but can support messages up to 2^64 bytes (2^32 segments of 4GB each), and it actually does allow you to read one field from the message without parsing the whole message (if it's in a file on disk, use mmap() to avoid reading the whole thing in).
(披露:我是 Cap'n Proto 以及大部分 Google 开源 Protobuf 代码的作者.)
(Disclosure: I'm the author of Cap'n Proto as well as most of Google's open source Protobuf code.)
这篇关于google protobuf 最大大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
