如何读取大型平面文件 [英] How do I read in a large flat file

问题描述

我有一个平面文件，其中包含339276行文本，大小为62.1 MB.我试图读取所有行，根据我所具有的某些条件解析它们，然后将它们插入数据库中.

I have a flat file that has 339276 line of text in it for a size of 62.1 MB. I am attempting to read in all the lines, parse them based on some conditions I have and then insert them into a database.

我最初尝试使用bufio.Scan()循环和bufio.Text()来获取行，但我的缓冲区空间不足.我切换到使用bufio.ReadLine/ReadString/ReadByte(我尝试了每种方法)，并且每种方法都遇到了相同的问题.我没有足够的缓冲区空间.

I originally attempted to use a bufio.Scan() loop and bufio.Text() to get the line but I was running out of buffer space. I switched to using bufio.ReadLine/ReadString/ReadByte (I tried each) and had the same problem with each. I didn't have enough buffer space.

我尝试使用读取并设置缓冲区大小，但是正如文档所述，它实际上是一个常量，可以使其变小，但不能大于64 * 1024字节.然后，我尝试使用File.ReadAt来设置起始姿势，并将其移动到每节中都没有用.我查看了以下示例和说明(不是详尽的列表):

I tried using read and setting the buffer size but as the document says it actually a const that can be made smaller but never bigger that 64*1024 bytes. I then tried to use File.ReadAt where I set the starting postilion and moved it along as I brought in each section to no avail. I have looked at the following examples and explanations (not an exhaustive list):

将文本文件读入字符串数组(并写入) 如何阅读最后每10秒通过Go扫描一个大文件中的行数 逐行读取文件

如何将整个文件(逐行或一次读取)切成片，然后再对行进行处理?

How do I read in an entire file (either line by line or the whole thing at once) into a slice so I can then go do things to the lines?

这是我尝试过的一些代码:

Here is some code that I have tried:

                 file, err := os.Open(feedFolder + value)
                 handleError(err)
                 defer file.Close()
                 //              fileInfo, _ := file.Stat()
                 var linesInFile []string

             r := bufio.NewReader(file)
             for {
                     path, err := r.ReadLine("\n") // 0x0A separator = newline

                     linesInFile = append(linesInFile, path)
                     if err == io.EOF {
                             fmt.Printf("End Of File: %s", err)
                             break
                     } else if err != nil {
                             handleError(err) // if you return error
                     }
             }
             fmt.Println("Last Line: ", linesInFile[len(linesInFile)-1])

这是我尝试过的其他东西:

Here is something else I tried:

var fileSize int64 = fileInfo.Size()
    fmt.Printf("File Size: %d\t", fileSize)
    var bufferSize int64 = 1024 * 60
    bytes := make([]byte, bufferSize)
    var fullFile []byte
    var start int64 = 0
    var interationCounter int64 = 1
    var currentErr error = nil
         for currentErr != io.EOF {
            _, currentErr = file.ReadAt(bytes, st)
            fullFile = append(fullFile, bytes...)
            start = (bufferSize * interationCounter) + 1
            interationCounter++
          }
     fmt.Printf("Err: %s\n", currentErr)
     fmt.Printf("fullFile Size: %s\n", len(fullFile))
     fmt.Printf("Start: %d", start)

     var currentLine []string


   for _, value := range fullFile {
      if string(value) != "\n" {
          currentLine = append(currentLine, string(value))
      } else {
         singleLine := strings.Join(currentLine, "")
         linesInFile = append(linesInFile, singleLine)
         currentLine = nil
              }   
      }

我很茫然.我要么不完全了解缓冲区的工作方式，要么不了解其他内容.谢谢阅读.

I am at a loss. Either I don't understand exactly how the buffer works or I don't understand something else. Thanks for reading.

推荐答案

bufio.Scan()和bufio.Text()在循环中对大小更大的文件非常适合我，因此我想您的行数超出了缓冲区容量.然后

bufio.Scan() and bufio.Text() in a loop perfectly works for me on a files with much larger size, so I suppose you have lines exceeded buffer capacity. Then

• 检查行尾
• 以及您使用的哪个Go版本path, err :=r.ReadLine("\n") // 0x0A separator = newline?看起来func (b *bufio.Reader) ReadLine() (line []byte, isPrefix bool, err error)具有专门用于您的用例的返回值isPrefix http://golang.org/pkg/bufio/#Reader.ReadLine

• check your line ending
• and which Go version you use path, err :=r.ReadLine("\n") // 0x0A separator = newline? Looks like func (b *bufio.Reader) ReadLine() (line []byte, isPrefix bool, err error) has return value isPrefix specifically for your use case http://golang.org/pkg/bufio/#Reader.ReadLine

这篇关于如何读取大型平面文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！