Golang-如何克服bufio的Scan（）缓冲区限制？ [英] Golang - How to overcome Scan() buffer limit from bufio?

问题描述

我正在使用 bufio 扫描仪的 .Scan（）方法读取文本文件行。但是，当我在文件行中达到一定大小时，扫描仪不再允许我读取它，而是返回一个空行。

I am using the bufio scanner's .Scan() method to read text file lines. However when I reach a certain size in the file lines, the scanner no longer allows me to read, it just returns an empty line.

我应该如何配置缓冲区以

How should I configure the buffer to take in larger amounts of data?

bigfile.txt 只是一个文件，其中许多行之间用空间。例如，一行中有40000个整数。 （请注意，它适用于10000个以下整数的文件行，但不适用于40000个文件行）

bigfile.txt is just a file with many integers in a line separated by a space. For example, 40000 integers in one line. (note that it works for file lines of 10000 integers and less, but not 40000 for example)

234 544 765 45 34 67 67 87 98 43 [... n = 40000]

234 544 765 45 34 67 67 87 98 43 [... n = 40000 ]

func main() {
    readInputFile("bigfile.txt")
}

func readInputFile(name string) {
    inFile, _ := os.Open(name)
    defer inFile.Close()

    scanner := bufio.NewScanner(inFile)

    for scanner.Scan() {
        line := scanner.Text()
        fmt.Printf(line)
    }
}

推荐答案

根据文档和相应的源代码，默认情况下，扫描仪使用内部缓冲区，容量为64K。在您的情况下，内部缓冲区不足以存储40000个整数。设置扫描器在调用 Scan 之前使用的缓冲区，即

According to documentation and corresponding source code, by default Scanner uses internal buffer in which the capacity is 64K. In your case, the internal buffer is not sufficient for storing 40000 integers. Set the buffer to be used by scanner before calling Scan, i.e.

scanner := bufio.NewScanner(inFile)

//adjust the capacity to your need (max characters in line)
const maxCapacity = 512*1024  
buf := make([]byte, maxCapacity)
scanner.Buffer(buf, maxCapacity)

这篇关于Golang-如何克服bufio的Scan（）缓冲区限制？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！