如何读取一个大型的平面文件?

huangapple go评论84阅读模式
英文:

How do I read in a large flat file

问题

我有一个包含339276行文本的平面文件,大小为62.1 MB。我试图读取所有行,根据一些条件进行解析,然后将它们插入到数据库中。

我最初尝试使用bufio.Scan()循环和bufio.Text()来获取行,但是我遇到了缓冲区空间不足的问题。我尝试使用bufio.ReadLine/ReadString/ReadByte(我尝试了每种方法),但是每种方法都遇到了同样的问题,缓冲区空间不足。

我尝试使用read并设置缓冲区大小,但是文档中说它实际上是一个常量,可以变得更小,但不能超过64*1024字节。然后我尝试使用File.ReadAt,在每次读取每个部分时设置起始位置并将其移动,但是没有成功。我查看了以下示例和解释(不是详尽无遗的列表):

https://stackoverflow.com/questions/5884154/golang-read-text-file-into-string-array-and-write
https://stackoverflow.com/questions/17863821/how-to-read-last-lines-from-a-big-file-with-go-every-10-secs
https://stackoverflow.com/questions/8757389/reading-file-line-by-line-in-go

我该如何将整个文件(逐行或一次性)读入切片中,以便我可以对行进行操作?

这是我尝试过的一些代码:

file, err := os.Open(feedFolder + value)
handleError(err)
defer file.Close()
var linesInFile []string

r := bufio.NewReader(file)
for {
    path, err := r.ReadLine("\n") // 0x0A separator = newline

    linesInFile = append(linesInFile, path)
    if err == io.EOF {
        fmt.Printf("End Of File: %s", err)
        break
    } else if err != nil {
        handleError(err) // if you return error
    }
}
fmt.Println("Last Line: ", linesInFile[len(linesInFile)-1])

这是我尝试过的另一种方法:

var fileSize int64 = fileInfo.Size()
fmt.Printf("File Size: %d\t", fileSize)
var bufferSize int64 = 1024 * 60
bytes := make([]byte, bufferSize)
var fullFile []byte
var start int64 = 0
var interationCounter int64 = 1
var currentErr error = nil
for currentErr != io.EOF {
    _, currentErr = file.ReadAt(bytes, st)
    fullFile = append(fullFile, bytes...)
    start = (bufferSize * interationCounter) + 1
    interationCounter++
}
fmt.Printf("Err: %s\n", currentErr)
fmt.Printf("fullFile Size: %s\n", len(fullFile))
fmt.Printf("Start: %d", start)

var currentLine []string

for _, value := range fullFile {
    if string(value) != "\n" {
        currentLine = append(currentLine, string(value))
    } else {
        singleLine := strings.Join(currentLine, "")
        linesInFile = append(linesInFile, singleLine)
        currentLine = nil
    }
}

我感到困惑。要么我不完全理解缓冲区的工作原理,要么我不理解其他一些东西。谢谢阅读。

英文:

I have a flat file that has 339276 line of text in it for a size of 62.1 MB. I am attempting to read in all the lines, parse them based on some conditions I have and then insert them into a database.

I originally attempted to use a bufio.Scan() loop and bufio.Text() to get the line but I was running out of buffer space. I switched to using bufio.ReadLine/ReadString/ReadByte (I tried each) and had the same problem with each. I didn't have enough buffer space.

I tried using read and setting the buffer size but as the document says it actually a const that can be made smaller but never bigger that 64*1024 bytes. I then tried to use File.ReadAt where I set the starting postilion and moved it along as I brought in each section to no avail. I have looked at the following examples and explanations (not an exhaustive list):

https://stackoverflow.com/questions/5884154/golang-read-text-file-into-string-array-and-write
https://stackoverflow.com/questions/17863821/how-to-read-last-lines-from-a-big-file-with-go-every-10-secs
https://stackoverflow.com/questions/8757389/reading-file-line-by-line-in-go

How do I read in an entire file (either line by line or the whole thing at once) into a slice so I can then go do things to the lines?

Here is some code that I have tried:

                 file, err := os.Open(feedFolder + value)
                 handleError(err)
                 defer file.Close()
                 //              fileInfo, _ := file.Stat()
                 var linesInFile []string

             r := bufio.NewReader(file)
             for {
                     path, err := r.ReadLine("\n") // 0x0A separator = newline

                     linesInFile = append(linesInFile, path)
                     if err == io.EOF {
                             fmt.Printf("End Of File: %s", err)
                             break
                     } else if err != nil {
                             handleError(err) // if you return error
                     }
             }
             fmt.Println("Last Line: ", linesInFile[len(linesInFile)-1])

Here is something else I tried:

var fileSize int64 = fileInfo.Size()
    fmt.Printf("File Size: %d\t", fileSize)
    var bufferSize int64 = 1024 * 60
    bytes := make([]byte, bufferSize)
    var fullFile []byte
    var start int64 = 0
    var interationCounter int64 = 1
    var currentErr error = nil
         for currentErr != io.EOF {
            _, currentErr = file.ReadAt(bytes, st)
            fullFile = append(fullFile, bytes...)
            start = (bufferSize * interationCounter) + 1
            interationCounter++
          }
     fmt.Printf("Err: %s\n", currentErr)
     fmt.Printf("fullFile Size: %s\n", len(fullFile))
     fmt.Printf("Start: %d", start)

     var currentLine []string
       
       
   for _, value := range fullFile {
      if string(value) != "\n" {
          currentLine = append(currentLine, string(value))
      } else {
         singleLine := strings.Join(currentLine, "")
         linesInFile = append(linesInFile, singleLine)
         currentLine = nil
              }   
      }

I am at a loss. Either I don't understand exactly how the buffer works or I don't understand something else. Thanks for reading.

答案1

得分: 8

bufio.Scan()bufio.Text()在一个循环中对于我来说完美地适用于更大的文件,所以我猜你的行超过了缓冲区的容量。然后:

  • 检查你的行结束符
  • 以及你使用的Go版本path, err := r.ReadLine("\n") // 0x0A分隔符 = 换行?看起来func (b *bufio.Reader) ReadLine() (line []byte, isPrefix bool, err error)有一个返回值isPrefix特别适用于你的用例
    http://golang.org/pkg/bufio/#Reader.ReadLine
英文:

bufio.Scan() and bufio.Text() in a loop perfectly works for me on a files with much larger size, so I suppose you have lines exceeded buffer capacity. Then

  • check your line ending
  • and which Go version you use path, err :=r.ReadLine("\n") // 0x0A separator = newline? Looks like func (b *bufio.Reader) ReadLine() (line []byte, isPrefix bool, err error) has return value isPrefix specifically for your use case
    http://golang.org/pkg/bufio/#Reader.ReadLine

答案2

得分: 6

不清楚是否有必要在解析并将其插入数据库之前读取所有行。尽量避免这样做。

你有一个小文件:“一个大小为62.1 MB的文本文件,其中包含339276行文本。”例如,

package main

import (
	"bytes"
	"fmt"
	"io"
	"io/ioutil"
)

func readLines(filename string) ([]string, error) {
	var lines []string
	file, err := ioutil.ReadFile(filename)
	if err != nil {
		return lines, err
	}
	buf := bytes.NewBuffer(file)
	for {
		line, err := buf.ReadString('\n')
		if len(line) == 0 {
			if err != nil {
				if err == io.EOF {
					break
				}
				return lines, err
			}
		}
		lines = append(lines, line)
		if err != nil && err != io.EOF {
			return lines, err
		}
	}
	return lines, nil
}

func main() {
	// 一个大小为62.1 MB的文本文件,其中包含339276行文本
	filename := "flat.file"
	lines, err := readLines(filename)
	fmt.Println(len(lines))
	if err != nil {
		fmt.Println(err)
		return
	}
}
英文:

It's not clear that it's necessary to read in all the lines before parsing them and inserting them into a database. Try to avoid that.

You have a small file: "a flat file that has 339276 line of text in it for a size of 62.1 MB." For example,

package main

import (
	"bytes"
	"fmt"
	"io"
	"io/ioutil"
)

func readLines(filename string) ([]string, error) {
	var lines []string
	file, err := ioutil.ReadFile(filename)
	if err != nil {
		return lines, err
	}
	buf := bytes.NewBuffer(file)
	for {
		line, err := buf.ReadString('\n')
		if len(line) == 0 {
			if err != nil {
				if err == io.EOF {
					break
				}
				return lines, err
			}
		}
		lines = append(lines, line)
		if err != nil && err != io.EOF {
			return lines, err
		}
	}
	return lines, nil
}

func main() {
	// a flat file that has 339276 lines of text in it for a size of 62.1 MB
	filename := "flat.file"
	lines, err := readLines(filename)
	fmt.Println(len(lines))
	if err != nil {
		fmt.Println(err)
		return
	}
}

答案3

得分: 2

这是我翻译好的内容:

我觉得这个readLines的变体比peterSO建议的更短更快。

func readLines(filename string) (map[int]string, error) {
    lines := make(map[int]string)

    data, err := ioutil.ReadFile(filename)
    if err != nil {
        return nil, err
    }

    for n, line := range strings.Split(string(data), "\n") {
        lines[n] = line
    }

    return lines, nil
}
英文:

It seems to me this variant of readLines is shorter and faster than suggested peterSO

func readLines(filename string) (map[int]string, error) {
	lines := make(map[int]string)

	data, err := ioutil.ReadFile(filename)
	if err != nil {
		return nil, err
	}

	for n, line := range strings.Split(string(data), "\n") {
		lines[n] = line
	}

	return lines, nil
}

答案4

得分: -1

package main

import (
    "fmt"
    "os"
    "log"
    "bufio"
)

func main() {
    FileName := "assets/file.txt"
    file, err := os.Open(FileName)
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)

    for scanner.Scan() { 
        fmt.Println(scanner.Text()) 

    }
}

这是一个Go语言的代码片段,它打开一个名为"assets/file.txt"的文件,并逐行读取文件内容并打印出来。

英文:
package main

import (
    "fmt"
    "os"
    "log"
    "bufio"
)

func main() {
    FileName := "assets/file.txt"
	file, err := os.Open(FileName)
	if err != nil {
		log.Fatal(err)
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)

	for scanner.Scan() { 
		fmt.Println(scanner.Text()) 

	}
}

huangapple
  • 本文由 发表于 2015年3月28日 10:54:42
  • 转载请务必保留本文链接:https://go.coder-hub.com/29313133.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定