How to get file length in Go dynamically?

huangapple
go评论105阅读模式
英文:

How to get file length in Go dynamically?

问题

我有以下代码片段：

  1. func main() {
  2.     // Some text we want to compress.
  3.     original := "bird and frog"
  4.     
  5.     // Open a file for writing.
  6.     f, _ := os.Create("C:\\programs\\file.gz")
  7.     
  8.     // Create gzip writer.
  9.     w := gzip.NewWriter(f)
  10.     
  11.     // Write bytes in compressed form to the file.
  12.     for i := 0; i < 10; i++ {
  13.         // Get the row from the database as obtained from cursor.
  14.         row := []byte("row from the database")
  15.         
  16.         // Compress the row.
  17.         compressedRow := compress(row)
  18.         
  19.         // Write the compressed row to the file.
  20.         w.Write(compressedRow)
  21.         
  22.         // Check if the file size exceeds the limit.
  23.         if f.Size() > 50 {
  24.             // Close the current file.
  25.             w.Close()
  26.             f.Close()
  27.             
  28.             // Open a new file for writing.
  29.             f, _ = os.Create("C:\\programs\\file.gz")
  30.             
  31.             // Create a new gzip writer.
  32.             w = gzip.NewWriter(f)
  33.         }
  34.     }
  35.     
  36.     // Close the file.
  37.     w.Close()
  38.     f.Close()
  39.     
  40.     fmt.Println("DONE")
  41. }
  43. func compress(data []byte) []byte {
  44.     // Compress the data using gzip compression algorithm.
  45.     // Return the compressed data.
  46.     return data
  47. }

这段代码实现了在文件大小达到一定阈值时关闭当前文件并打开一个新文件的功能。在每次写入压缩文档之前，通过检查文件大小是否超过限制来判断是否需要关闭当前文件并打开一个新文件。你可以根据实际需求修改代码中的数据库行获取逻辑和压缩逻辑。

英文:

I have the following code snippet:

  1. func main() {
  2.     // Some text we want to compress.
  3.     original := &quot;bird and frog&quot;
  4.     
  5.     // Open a file for writing.
  6.     f, _ := os.Create(&quot;C:\\programs\\file.gz&quot;)
  7.     
  8.     // Create gzip writer.
  9.     w := gzip.NewWriter(f)
  10.     
  11.     // Write bytes in compressed form to the file.
  12.     while ( looping over database cursor) {
  13.        w.Write([]byte(/* the row from the database as obtained from cursor */))
  14.     }
  15.     
  16.     // Close the file.
  17.     w.Close()
  18.     
  19.     fmt.Println(&quot;DONE&quot;)
  20. }

However, I wish to know a small modification. When the size of file reaches a certain threshold I want to close it and open a new file. And that too in compressed format.

For example:

Assume a database has 10 rows each row is 50 bytes.

Assume compression factor is 2, ie 1 row of 50 bytes is compressed to 25 bytes.

Assume a file size limit is 50 bytes.

Which means after every 2 records I should close the file and open a new file.

How to keep track of the file size while its still open and still writing compressed documents to it ?

答案1

得分: 2

gzip.NewWriter 接受一个 io.Writer。很容易实现自定义的 io.Writer 来满足你的需求。

例如：Playground

  1. type MultiFileWriter struct {
  2.     maxLimit      int
  3.     currentSize   int
  4.     currentWriter io.Writer
  5. }
  7. func (m *MultiFileWriter) Write(data []byte) (n int, err error) {
  8.     if len(data)+m.currentSize > m.maxLimit {
  9.         m.currentWriter = createNextFile()
  10.     }
  11.     m.currentSize += len(data)
  12.     return m.currentWriter.Write(data)
  13. }

注意：你需要处理一些边界情况，比如如果 len(data) 大于 maxLimit 会怎样。也许你不想将一条记录拆分到多个文件中。

英文:

gzip.NewWriter takes a io.Writer. It is easy to implement custom io.Writer that does what you want.

E.g. Playground

  1. type MultiFileWriter struct {
  2. 	maxLimit      int
  3. 	currentSize   int
  4. 	currentWriter io.Writer
  5. }
  7. func (m *MultiFileWriter) Write(data []byte) (n int, err error) {
  8. 	if len(data)+m.currentSize &gt; m.maxLimit {
  9. 		m.currentWriter = createNextFile()
  10. 	}
  11.     m.currentSize += len(data)
  12. 	return m.currentWriter.Write(data)
  13. }

Note: You will need to handle few edge cases like what if len(data) is greater than the maxLimit. And may be you don't want to split a record across files.

答案2

得分: 1

你可以使用os.File.Seek方法来获取文件中的当前位置，因为你正在写入文件，所以当前位置将是文件的当前大小（以字节为单位）。

例如：

  1. package main
  3. import (
  4. 	"compress/gzip"
  5. 	"fmt"
  6. 	"os"
  7. )
  9. func main() {
  10. 	// 要压缩的文本。
  11. 	lines := []string{
  12. 		"this is a test",
  13. 		"the quick brown fox",
  14. 		"jumped over the lazy dog",
  15. 		"the end",
  16. 	}
  18. 	// 打开一个文件进行写入。
  19. 	f, err := os.Create("file.gz")
  20. 	if err != nil {
  21. 		panic(err)
  22. 	}
  24. 	// 创建gzip写入器。
  25. 	w := gzip.NewWriter(f)
  27. 	// 以压缩形式将字节写入文件。
  28. 	for _, line := range lines {
  29. 		w.Write([]byte(line))
  31. 		w.Flush()
  32. 		pos, err := f.Seek(0, os.SEEK_CUR)
  33. 		if err != nil {
  34. 			panic(err)
  35. 		}
  37. 		fmt.Printf("pos: %d\n", pos)
  38. 	}
  40. 	// 关闭文件。
  41. 	w.Close()
  43. 	// 调用w.Close()将写出任何剩余的数据和最终的校验和。
  44. 	pos, err := f.Seek(0, os.SEEK_CUR)
  45. 	if err != nil {
  46. 		panic(err)
  47. 	}
  48. 	fmt.Printf("pos: %d\n", pos)
  50. 	fmt.Println("DONE")
  51. }

输出结果为：

  1. pos: 30
  2. pos: 55
  3. pos: 83
  4. pos: 94
  5. pos: 107
  6. DONE

我们可以使用wc命令进行确认：

  1. $ wc -c file.gz
  2. 107 file.gz
英文:

You can use the os.File.Seek method to get your current position in the file, which as you're writing the file will be the current file size in bytes.

For example:

  1. package main
  3. import (
  4. 	&quot;compress/gzip&quot;
  5. 	&quot;fmt&quot;
  6. 	&quot;os&quot;
  7. )
  9. func main() {
  10. 	// Some text we want to compress.
  11. 	lines := []string{
  12. 		&quot;this is a test&quot;,
  13. 		&quot;the quick brown fox&quot;,
  14. 		&quot;jumped over the lazy dog&quot;,
  15. 		&quot;the end&quot;,
  16. 	}
  18. 	// Open a file for writing.
  19. 	f, err := os.Create(&quot;file.gz&quot;)
  20. 	if err != nil {
  21. 		panic(err)
  22. 	}
  24. 	// Create gzip writer.
  25. 	w := gzip.NewWriter(f)
  27. 	// Write bytes in compressed form to the file.
  28. 	for _, line := range lines {
  29. 		w.Write([]byte(line))
  31. 		w.Flush()
  32. 		pos, err := f.Seek(0, os.SEEK_CUR)
  33. 		if err != nil {
  34. 			panic(err)
  35. 		}
  37. 		fmt.Printf(&quot;pos: %d\n&quot;, pos)
  38. 	}
  40. 	// Close the file.
  41. 	w.Close()
  43. 	// The call to w.Close() will write out any remaining data
  44. 	// and the final checksum.
  45. 	pos, err := f.Seek(0, os.SEEK_CUR)
  46. 	if err != nil {
  47. 		panic(err)
  48. 	}
  49. 	fmt.Printf(&quot;pos: %d\n&quot;, pos)
  51. 	fmt.Println(&quot;DONE&quot;)
  52. }

Which outputs:

  1. pos: 30
  2. pos: 55
  3. pos: 83
  4. pos: 94
  5. pos: 107
  6. DONE

And we can confirm with wc:

  1. $ wc -c file.gz
  2. 107 file.gz

huangapple
  • 本文由 发表于 2022年9月14日 06:42:26
  • 转载请务必保留本文链接：https://go.coder-hub.com/73709794.html
  • file
  • go
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定