通过TCP在python中接收分隔的Protobuf消息 [英] Receive delimited Protobuf message in python via TCP

问题描述

我正在尝试接收 protobuf 消息，该消息是从我的 Python 应用程序中带有writeDelmitedTo()"的 Java 应用程序发送的.

I am trying to receive a protobuf message, which was send from a java application with "writeDelmitedTo()" inside my python application.

经过一些研究，我已经遇到了这段代码，可以从套接字读取消息，对其进行解码和解析.

After some research I already came across this code to read the message from the socket, decode it and parse it.

data = sock.recv()
(size, position) = decoder._DecodeVarint(data, 0)
msg = MessageWrapper_pb2.WrapperMessage().ParseFromString(data[position:position + size])

我现在得到的是 google.protobuf.message.DecodeError: Truncated message Exception.

What I am getting now is a google.protobuf.message.DecodeError: Truncated message Exception.

有没有人遇到过类似的问题或者知道如何从套接字读取分隔数据并正确解析它?

Has anyone encountered a similar problem or knows how to read delimited data from a socket and parse it correctly?

这是对我有用的解决方案.

This is the solution that worked for me.

def read_java_varint_delimited_stream(sock):
    buf = []
    data = sock.recv()
    rCount = len(data)
    (size, position) = decoder._DecodeVarint(data, 0)

    buf.append(data)
    while rCount < size+position:
        data = sock.recv(size+position-rCount)
        rCount += len(data)
        buf.append(data)

    return b''.join(buf), size, position

def readMessage(sock):
    data, size, position = read_java_varint_delimited_stream(sock)
    msg = MessageWrapper_pb2.WrapperMessage()
    msg.ParseFromString(data[position:position + size])

    return msg

推荐答案

TCP 是一种流协议，并没有说一端的 recv 与单个 send<配对/code> 在另一端.基于消息的协议需要某种方式来标记其边界，以便接收方知道如何找到消息边界.

TCP is a stream protocol and there is nothing that says a recv on one end is paired with a single send on the other end. Message based protocols need some way to mark their boundaries so the receiver knows how to find message boundaries.

writeDelimitedTo docs 表示发送 varint 大小，然后发送数据.因此，读取 varint 然后读取该字节数.

The writeDelimitedTo docs say that a varint size is sent and then the data. So, read the varint and then read that number of bytes.

深入挖掘，varint 文档描述了它的价值通过使用字节的高位来标记连续性进行编码.我们可以编写自己的解码器

Digging deeper, the varint docs describe how its value is encoded by using a byte's high-bit to mark continuation. We we can write up our own decoder

import struct

def read_java_varint_delimited_stream(sock):
    sz = 0
    while True:
        vbyte, = struct.unpack('b', sock.recv(1))
        sz = (vbyte << 7) + (vbyte & 0x7f)
        if not vbyte & 0x80:
            break
    data = []
    while sz:
        buf = sock.recv(sz)
        if not buf:
            raise ValueError("Buffer receive truncated")
        data.append(buf)
        sz -= len(buf)
    return b''.join(buf)

这篇关于通过TCP在python中接收分隔的Protobuf消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！