记录未知长度参数的大小控制 [英] Size control on logging an unknown length of parameters

问题描述

问题:

现在，我正在记录我的SQL查询和与该查询相关的参数，但是如果我的参数很重，会发生什么?说100MB?

解决方案:

我想遍历args，一旦它们超过了0.5MB，我要处理args直到现在并只记录它们(当然，我将使用实际SQL查询中设置的整个args)./p>

卡住的地方:

1. 我发现很难找到接口{} 的磁盘大小.
2. 如何打印?(有一种比％v 更好的方法吗?)

关注主要集中在第一部分，如何找到大小，我需要知道类型，如果它是数组，堆栈，堆等.

如果代码有帮助，这是我的代码结构(所有文件都位于util文件中的dal pkg中):

  package dal进口 ("fmt")const limitedLogArgsSizeB = 100000//〜0.1MBfunc parsedArgs(args ... interface {})字符串{currentSize:= 0var res字符串对于我:= 0;我<len(args);我++ {currentEleSize:= getSizeOfElement(args [i])如果！(currentSize + currentEleSize =< limitedLogArgsSizeB){休息}currentSize + = currentEleSizeres = fmt.Sprintf(％s，％v"，res，args [i])}返回"[" + res +]"}func getSizeOfElement(interface {})(sizeInBytes int){} 

因此，如您所见，我期望从parsedArgs()返回的字符串类似于:

"[[4378233，33，是]"

为完整起见，随之而来的查询:

  INSERT INTO Person(id，age，is_healthy)VALUES($ 0，$ 1，$ 2) 

所以要说明所有这些点:

让我们说前两个参数完全等于我要记录的大小限制的阈值，我只会从parsedArgs()中将前两个参数作为字符串返回，如下所示:

"[4378233，33]"

我可以根据要求提供更多详细信息，谢谢:)

解决方案

获取任意值(任意数据结构)的内存大小并非不可能，但在Go中是困难的".有关详细信息，请参见如何获取Go中的变量?

最简单的解决方案可能是生成要记录在内存中的数据，您可以在记录之前简单地截断它(例如，如果它是 string 或字节切片，则只需对其进行切片).但是，这不是最温和的解决方案(速度较慢，需要更多内存).

相反，我会以不同的方式实现您想要的.我将尝试组装要记录的数据，但我将使用特殊的 io.Writer 作为目标(可能针对您的磁盘或内存缓冲区)，以跟踪写入的字节数，一旦达到限制，它可能会丢弃更多数据(或报告错误，无论是否适合您).

您可以在此处查看计数中的 io.Writer 实现:

这将输出(在游乐场上尝试):

  20"[1 2 Looooooooooooon" 

如您所见，我们的 LimitWriter 仅允许写入20个字节( LimitWriter.Remaining )，其余部分则被丢弃.

请注意，在此示例中，我将数据组装在内存缓冲区中，但是在您的日志记录系统中，您可以直接将其写入日志记录流，只需将其包装在 LimitWriter 中(这样您就可以完全省略内存缓冲区.

优化提示:如果您将参数作为切片，则可以使用循环来优化截断的渲染，并在达到限制后停止打印参数.

执行此操作的示例:

  buf:=& bytes.Buffer {}lw:=& LimitWriter {作家:buf，剩余:20个}args:= [] interface {} {1,2，"Loooooooooooooooong"，3，4，5}io.WriteString(lw，"[")对于我，v:=范围args {如果_，err:= fmt.Fprint(lw，v，");err！= nil {fmt.Printf(在参数％d处中断，错误:％v \ n"，i，错误)休息}}io.WriteString(lw，]")fmt.Printf(％d％q"，buf.Len()，buf) 

输出(在游乐场上尝试):

 突破参数3，错误:EOF20"[1 2 Loooooooooooooo" 

这样做的好处是，一旦达到极限，就不必生成其余参数的字符串表示形式了，这些参数无论如何都会被丢弃，从而节省了一些CPU(和内存)资源.

The Problem:

Right now, I'm logging my SQL query and the args that related to that query, but what will happen if my args weight a lot? say 100MB?

The Solution:

I want to iterate over the args and once they exceeded the 0.5MB I want to take the args up till this point and only log them (of course I'll use the entire args set in the actual SQL query).

Where am stuck:

1. I find it hard to find the size on the disk of an interface{}.
2. How can I print it? (there is a nicer way to do it than %v?)

The concern is mainly focused on the first section, how can I find the size, I need to know the type, if its an array, stack, heap, etc..

If code helps, here is my code structure (everything sits in dal pkg in util file):

package dal

import (
    "fmt"
)

const limitedLogArgsSizeB = 100000 // ~ 0.1MB

func parsedArgs(args ...interface{}) string {
    currentSize := 0
    var res string

    for i := 0; i < len(args); i++ {
        currentEleSize := getSizeOfElement(args[i])

        if !(currentSize+currentEleSize =< limitedLogArgsSizeB) {
            break
        }

        currentSize += currentEleSize

        res = fmt.Sprintf("%s, %v", res, args[i])
    }

    return "[" + res + "]"
}

func getSizeOfElement(interface{}) (sizeInBytes int) {

}

So as you can see I expect to get back from parsedArgs() a string that looks like:

"[4378233, 33, true]"

for completeness, the query that goes with it:

INSERT INTO Person (id,age,is_healthy) VALUES ($0,$1,$2)

so to demonstrate the point of all of this:

lets say the first two args are equal exactly to the threshold of the size limit that I want to log, I will only get back from the parsedArgs() the first two args as a string like this:

"[4378233, 33]"

I can provide further details upon request, Thanks :)

解决方案

Getting the memory size of arbitrary values (arbitrary data structures) is not impossible but "hard" in Go. For details, see How to get memory size of variable in Go?

The easiest solution could be to produce the data to be logged in memory, and you can simply truncate it before logging (e.g. if it's a string or a byte slice, simply slice it). This is however not the gentlest solution (slower and requires more memory).

Instead I would achieve what you want differently. I would try to assemble the data to be logged, but I would use a special io.Writer as the target (which may be targeted at your disk or at an in-memory buffer) which keeps track of the bytes written to it, and once a limit is reached, it could discard further data (or report an error, whatever suits you).

You can see a counting io.Writer implementation here: Size in bits of object encoded to JSON?

type CounterWr struct {
    io.Writer
    Count int
}

func (cw *CounterWr) Write(p []byte) (n int, err error) {
    n, err = cw.Writer.Write(p)
    cw.Count += n
    return
}

We can easily change it to become a functional limited-writer:

type LimitWriter struct {
    io.Writer
    Remaining int
}

func (lw *LimitWriter) Write(p []byte) (n int, err error) {
    if lw.Remaining == 0 {
        return 0, io.EOF
    }

    if lw.Remaining < len(p) {
        p = p[:lw.Remaining]
    }
    n, err = lw.Writer.Write(p)
    lw.Remaining -= n
    return
}

And you can use the fmt.FprintXXX() functions to write into a value of this LimitWriter.

An example writing to an in-memory buffer:

buf := &bytes.Buffer{}
lw := &LimitWriter{
    Writer:     buf,
    Remaining:  20,
}

args := []interface{}{1, 2, "Looooooooooooong"}

fmt.Fprint(lw, args)
fmt.Printf("%d %q", buf.Len(), buf)

This will output (try it on the Go Playground):

20 "[1 2 Looooooooooooon"

As you can see, our LimitWriter only allowed to write 20 bytes (LimitWriter.Remaining), and the rest were discarded.

Note that in this example I assembled the data in an in-memory buffer, but in your logging system you can write directly to your logging stream, just wrap it in LimitWriter (so you can completely omit the in-memory buffer).

Optimization tip: if you have the arguments as a slice, you may optimize the truncated rendering by using a loop, and stop printing arguments once the limit is reached.

An example doing this:

buf := &bytes.Buffer{}
lw := &LimitWriter{
    Writer:    buf,
    Remaining: 20,
}

args := []interface{}{1, 2, "Loooooooooooooooong", 3, 4, 5}

io.WriteString(lw, "[")
for i, v := range args {
    if _, err := fmt.Fprint(lw, v, " "); err != nil {
        fmt.Printf("Breaking at argument %d, err: %v\n", i, err)
        break
    }
}
io.WriteString(lw, "]")

fmt.Printf("%d %q", buf.Len(), buf)

Output (try it on the Go Playground):

Breaking at argument 3, err: EOF
20 "[1 2 Loooooooooooooo"

The good thing about this is that once we reach the limit, we don't have to produce the string representation of the remaining arguments that would be discarded anyway, saving some CPU (and memory) resources.

这篇关于记录未知长度参数的大小控制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！