如何在Golang中读取大型平面文件 [英] How do I read in a large flat file in Golang
问题描述
我有一个平面文件，其中包含339276行文本，大小为62.1 MB。我试图读取所有行，根据我所具有的某些条件解析它们，然后将它们插入数据库。
I have a flat file that has 339276 line of text in it for a size of 62.1 MB. I am attempting to read in all the lines, parse them based on some conditions I have and then insert them into a database.
我最初尝试使用bufio.Scan（）循环和bufio.Text（）来获取行，但我的缓冲区空间不足。我切换到使用bufio.ReadLine / ReadString / ReadByte（我尝试了每种方法），并且每种方法都有相同的问题。我没有足够的缓冲区空间。
I originally attempted to use a bufio.Scan() loop and bufio.Text() to get the line but I was running out of buffer space. I switched to using bufio.ReadLine/ReadString/ReadByte (I tried each) and had the same problem with each. I didn't have enough buffer space.
我尝试使用读取并设置缓冲区大小，但正如文档所述，它实际上是一个常量，可以使其变小，但不能大于64 * 1024字节。然后，我尝试使用File.ReadAt设置起始姿势并将其移动，因为我将每个部分都无济于事。我查看了以下示例和说明（不是详尽的列表）：
I tried using read and setting the buffer size but as the document says it actually a const that can be made smaller but never bigger that 64*1024 bytes. I then tried to use File.ReadAt where I set the starting postilion and moved it along as I brought in each section to no avail. I have looked at the following examples and explanations (not an exhaustive list):
将文本文件读入字符串数组（并写入）
如何使用Go每10秒从大文件读取最后几行
逐行读取文件
我如何将整个文件（逐行或一次读取）切成片，然后再对行进行处理？
How do I read in an entire file (either line by line or the whole thing at once) into a slice so I can then go do things to the lines?
这是我尝试过的一些代码：
Here is some code that I have tried:
file, err := os.Open(feedFolder + value)
handleError(err)
defer file.Close()
// fileInfo, _ := file.Stat()
var linesInFile []string
r := bufio.NewReader(file)
for {
path, err := r.ReadLine("\n") // 0x0A separator = newline
linesInFile = append(linesInFile, path)
if err == io.EOF {
fmt.Printf("End Of File: %s", err)
break
} else if err != nil {
handleError(err) // if you return error
}
}
fmt.Println("Last Line: ", linesInFile[len(linesInFile)-1])
这里是我尝试过的其他东西：
Here is something else I tried:
var fileSize int64 = fileInfo.Size()
fmt.Printf("File Size: %d\t", fileSize)
var bufferSize int64 = 1024 * 60
bytes := make([]byte, bufferSize)
var fullFile []byte
var start int64 = 0
var interationCounter int64 = 1
var currentErr error = nil
for currentErr != io.EOF {
_, currentErr = file.ReadAt(bytes, st)
fullFile = append(fullFile, bytes...)
start = (bufferSize * interationCounter) + 1
interationCounter++
}
fmt.Printf("Err: %s\n", currentErr)
fmt.Printf("fullFile Size: %s\n", len(fullFile))
fmt.Printf("Start: %d", start)
var currentLine []string
for _, value := range fullFile {
if string(value) != "\n" {
currentLine = append(currentLine, string(value))
} else {
singleLine := strings.Join(currentLine, "")
linesInFile = append(linesInFile, singleLine)
currentLine = nil
}
}
我在失利。我可能不完全了解缓冲区的工作方式，或者我不了解其他内容。谢谢阅读。
I am at a loss. Either I don't understand exactly how the buffer works or I don't understand something else. Thanks for reading.
推荐答案
bufio.Scan（）和 bufio.Text（）在一个循环中对我来说非常适用于更大的文件，所以我想您的行数超出了缓冲区容量。然后
bufio.Scan() and bufio.Text() in a loop perfectly works for me on a files with much larger size, so I suppose you have lines exceeded buffer capacity. Then
- 检查行尾
- 以及您使用的Go版本是
路径，错误：= r.ReadLine（ \n）// 0x0A分隔符=换行符？看起来像func（b * bufio.Reader）ReadLine（）（行[] byte，isPrefix bool，错误错误）具有返回值isPrefix专门针对您的用例
http://golang.org/ pkg / bufio /＃Reader.ReadLine
- check your line ending
- and which Go version you use
path, err :=r.ReadLine("\n") // 0x0A separator = newline? Looks likefunc (b *bufio.Reader) ReadLine() (line []byte, isPrefix bool, err error)has return valueisPrefixspecifically for your use case http://golang.org/pkg/bufio/#Reader.ReadLine
这篇关于如何在Golang中读取大型平面文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
