继续:有效处理通过TCP接收的碎片数据 [英] Go: efficiently handling fragmented data received via TCP

问题描述

我正在编写一个小型tcp服务器，该服务器仅用于读取数据，解析接收数据(动态长度帧)并处理这些帧.考虑以下代码:

I am writing a small tcp-server that is only suppose to read data, parse the receiving data (dynamic length frames) and handling these frames. Consider the following code:

func ClientHandler(conn net.Conn) {
    buf := make([]byte, 4096)
    n, err := conn.Read(buf)
    if err != nil {
        fmt.Println("Error reading:", err.Error())
        return
    }
    fmt.Println("received ", n, " bytes of data =", string(buf))
    handleData(buf)

这几乎是我代码的本质.我将X个字节读取到一个空缓冲区中，然后处理数据.

This is pretty much the essence of my code as is. I read X bytes into a empty buffer and then handle the data.

在以下情况下会发生问题:

The problem occurs when:

1. 一帧数据大于它试图从tcp连接读取的缓冲区.在这种情况下，只有收到剩余的数据后，我才能处理数据(但是接收的缓冲区已被先前的数据占用).
2. 当缓冲区包含一个半帧数据时，我需要处理第一帧，同时保留不完整的帧以备后用...在这种情况下，我需要从缓冲区中删除处理过的部分，并移动不完整的部分，以便为框架的其余部分留出空间.

两种情况都可能需要重新分配和复制数据，这可能是一项昂贵的操作?此外，除了扩展缓冲区外，我对如何处理大于缓冲区的帧一无所知……但是不断增加的缓冲区可能会导致性能问题和拒绝服务.最后但并非最不重要的一点是，我不了解Golang的标准库，不足以知道是否有任何显式构建的软件包可以处理这些情况.

Both scenarios would probably require reallocation and copying of data, which may perhaps be a costly operation? Furthermore, I have no ideas on how to handle frames that are larger than the buffer except for expanding the buffer... but an ever-increasing buffer may perhaps lead to performance issues and denial of service. Last, but not least, I do not know the standard library of Golang good enough to know if there are any package built explicitly for handling these situations.

所以我的问题是:

1. 是否有处理这些情况的最佳实践?
2. 是否有任何golang软件包可以为我完成部分或大部分任务?

谢谢.

推荐答案

字节片应支持非常优化的大小调整(即，保留比所需更多的字节，不复制，如果可以的话，成倍增长，复制代码不是用go编写的，而是运行时的一部分，等等).

byte slices should support pretty optimized resizing (i.e. reserving more bytes than needed, not copying if they can, growing exponentially, the copying code is not written in go but part of the runtime, etc).

因此您可以使用 append 并查看其工作原理.

so you can use append and see how it works.

另一种更惯用的方法是使用 bufio.Reader 来包装连接，它将自动处理所有这一切.我已经在我编写的tcp服务器中使用过它，而且速度非常快.您只要这样做:

Another, more idiomatic approach, is using bufio.Reader to wrap the connection, and it will handle all this automatically. I've used it in a tcp server I've written and it was extremely fast. You just do:

r := bufio.NewReader(conn)

，然后您可以读取直到分隔符或给定的字节数(如果您事先知道).

and then you can either read until a delimiter or a given amount of bytes if you know it in advance.

这篇关于继续:有效处理通过TCP接收的碎片数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！