使用goroutines构建Zip时，出现了切片边界超出范围的错误，容量为4096。

huangapple
go评论123阅读模式
英文:

Build Zip with goroutines panic with slice bounds out of range with capacity 4096

问题

我正在尝试改进一个函数，通过添加goroutines来处理每个需要归档的文件，以构建一个Zip文件。但是最终出现了一个panic错误：panic: runtime error: slice bounds out of range [4126:4096]。

目标目录包含190个文件（500 MB）。
我不太明白出了什么问题，
非常感谢您的帮助。

这是函数的代码：

  1. func BuildArchive() error {
  2.     var files []string
  3.     err := filepath.Walk("/tmp/dir-to-zip", func(filePath string, info os.FileInfo, err error) error {
  4.         if info.IsDir() {
  5.             return nil
  6.         }
  7.         if err != nil {
  8.             fmt.Println(err)
  9.             return err
  10.         }
  11.         files = append(files, filePath)
  13.         return nil
  14.     })
  15.     if err != nil {
  16.         return err
  17.     }
  19.     bundle, err := os.Create("/tmp/archive.zip")
  20.     if err != nil {
  21.         return err
  22.     }
  23.     bundleWriter := zip.NewWriter(bundle)
  25.     var wg sync.WaitGroup
  26.     wg.Add(len(files))
  28.     for _, filePath := range files {
  29.         go func(filePath string) {
  30.             defer wg.Done()
  31.             relPath := strings.TrimPrefix(filePath, fmt.Sprintf("%v/", filepath.Dir("/tmp/dir-to-zip")))
  33.             bundleFile, err := bundleWriter.Create(relPath)
  34.             if err != nil {
  35.                 fmt.Println(err)
  36.             }
  37.             fsFile, err := os.Open(filePath)
  38.             if err != nil {
  39.                 fmt.Println(err)
  40.             }
  41.             _, err = io.Copy(bundleFile, fsFile)
  42.             if err != nil {
  43.                 fmt.Println(err)
  44.             }
  46.         }(filePath)
  47.     }
  48.     wg.Wait()
  50.     err = bundleWriter.Close()
  51.     if err != nil {
  52.         return err
  53.     }
  54.     return nil
  55. }

错误发生在这里：

  1. _, err = io.Copy(bundleFile, fsFile)
  2. if err != nil {
  3.     fmt.Println(err)
  4. }

堆栈跟踪：

  1. goroutine 48 [running]:
  2. bufio.(*Writer).Write(0xc00002a100, {0xc00041a400?, 0x3d?, 0xc00041a400?})
  3.     /usr/local/go/src/bufio/bufio.go:670 +0x1c8
  4. archive/zip.(*countWriter).Write(0xc00000c138, {0xc00041a400?, 0x3d?, 0x4afa20?})
  5.     /usr/local/go/src/archive/zip/writer.go:601 +0x2e
  6. io.WriteString({0x4e7538, 0xc00000c138}, {0xc0000212c9, 0x3d})
  7.     /usr/local/go/src/io/io.go:314 +0x91
  8. archive/zip.writeHeader({0x4e7538, 0xc00000c138}, 0xc000220090)
  9.     /usr/local/go/src/archive/zip/writer.go:422 +0x5ec
  10. archive/zip.(*Writer).CreateHeader(0xc0000760a0, 0xc00021e1b0)
  11.     /usr/local/go/src/archive/zip/writer.go:378 +0x797
  12. archive/zip.(*Writer).Create(0x4e7698?, {0xc0000212c9, 0x3d})
  13.     /usr/local/go/src/archive/zip/writer.go:223 +0x6c
  14. main.BuildArchive.func2({0xc0000212c0, 0x46})
  15.     /home/simba/go/src/foobar/main.go:79 +0x1c5
  16. created by main.BuildArchive
  17.     /home/simba/go/src/foobar/main.go:73 +0x5aa
  18. panic: runtime error: slice bounds out of range [:4126] with capacity 4096
  20. goroutine 6 [running]:
  21. bufio.(*Writer).Flush(0xc00002a100)
  22.     /usr/local/go/src/bufio/bufio.go:634 +0x171
  23. bufio.(*Writer).Write(0xc00002a100, {0xc0001b4200?, 0xc000199b20?, 0xc000199b20?})
  24.     /usr/local/go/src/bufio/bufio.go:672 +0xd8
  25. archive/zip.(*countWriter).Write(0xc00000c138, {0xc0001b4200?, 0x0?, 0xc000199b40?})
  26.     /usr/local/go/src/archive/zip/writer.go:601 +0x2e
  27. archive/zip.(*countWriter).Write(0xc000220018, {0xc0001b4200?, 0xc0001b02c0?, 0xc0001b02f0?})
  28.     /usr/local/go/src/archive/zip/writer.go:601 +0x2e
  29. compress/flate.(*huffmanBitWriter).write(...)
  30.     /usr/local/go/src/compress/flate/huffman_bit_writer.go:136
  31. compress/flate.(*huffmanBitWriter).writeCode(0xc0001b41e0?, {0x6000?, 0x22?})
  32.     /usr/local/go/src/compress/flate/huffman_bit_writer.go:347 +0xe5
  33. compress/flate.(*huffmanBitWriter).writeTokens(0xc0001b41e0, {0xc002558000, 0x4001, 0x403f800000403f?}, {0xc0001aa900, 0x11e, 0x108129000000000f?}, {0xc0001ac100, 0x1e, 0x1e})
  34.     /usr/local/go/src/compress/flate/huffman_bit_writer.go:583 +0xb9
  35. compress/flate.(*huffmanBitWriter).writeBlock(0xc0001b41e0, {0xc002558000?, 0x20?, 0xd79?}, 0x0, {0x0, 0x0, 0x0})
  36.     /usr/local/go/src/compress/flate/huffman_bit_writer.go:495 +0x490
  37. compress/flate.(*compressor).writeBlock(0xc0005a2000, {0xc002558000?, 0xc000032f00?, 0xc000199d28?}, 0x47739b?)
  38.     /usr/local/go/src/compress/flate/deflate.go:170 +0x9c
  39. compress/flate.(*compressor).deflate(0xc0005a2000)
  40.     /usr/local/go/src/compress/flate/deflate.go:509 +0x59b
  41. compress/flate.(*compressor).write(0xc0005a2000, {0xc00256a000?, 0x8000, 0xf311b6fd?})
  42.     /usr/local/go/src/compress/flate/deflate.go:554 +0x82
  43. compress/flate.(*Writer).Write(...)
  44.     /usr/local/go/src/compress/flate/deflate.go:712
  45. archive/zip.(*pooledFlateWriter).Write(0xc00020c040?, {0xc00256a000?, 0x8000?, 0x4af140?})
  46.     /usr/local/go/src/archive/zip/register.go:51 +0xc5
  47. archive/zip.(*countWriter).Write(...)
  48.     /usr/local/go/src/archive/zip/writer.go:601
  49. archive/zip.(*fileWriter).Write(0xc000222000, {0xc00256a000, 0x8000, 0x8000})
  50.     /usr/local/go/src/archive/zip/writer.go:533 +0x97
  51. io.copyBuffer({0x4e7558, 0xc000222000}, {0x4e7678, 0xc0001f8008}, {0x0, 0x0, 0x0})
  52.     /usr/local/go/src/io/io.go:428 +0x204
  53. io.Copy(...)
  54.     /usr/local/go/src/io/io.go:385
  55. main.BuildArchive.func2({0xc00001c0c0, 0x35})
  56.     /home/simba/go/src/foobar/main.go:89 +0x385
  57. created by main.BuildArchive
  58.     /home/simba/go/src/foobar/main.go:73 +0x5aa
  59. exit status 2

希望这能帮到你！

英文:

I'm trying to improve a function to build a Zip by adding goroutines to handle each file that has to be archived.
But it ends up with a panic
panic: runtime error: slice bounds out of range [4126:4096]

The target directory contains 190 files (500 Mo).
I don't really understand what's wrong,
Thank you in advance for your help

The function :

  1. func BuildArchive() error {                                                        
  2. var files []string                                                             
  3. err := filepath.Walk("/tmp/dir-to-zip", func(filePath string, info os.FileInfo, err error) error {
  4. if info.IsDir() {                                                          
  5. return nil                                                             
  6. }                                                                          
  7. if err != nil {                                                            
  8. fmt.Println(err)                                                       
  9. return err                                                             
  10. }                                                                          
  11. files = append(files, filePath)                                            
  12. return nil                                                                 
  13. })                                                                             
  14. if err != nil {                                                                
  15. return err                                                                 
  16. }                                                                              
  17. bundle, err := os.Create("/tmp/archive.zip")                                   
  18. if err != nil {                                                                
  19. return err                                                                 
  20. }                                                                              
  21. bundleWriter := zip.NewWriter(bundle)                                          
  22. var wg sync.WaitGroup                                                          
  23. wg.Add(len(files))                                                             
  24. for _, filePath := range files {                                               
  25. go func(filePath string) {                                                 
  26. defer wg.Done()                                                        
  27. relPath := strings.TrimPrefix(filePath, fmt.Sprintf("%v/", filepath.Dir("/tmp/dir-to-zip")))
  28. bundleFile, err := bundleWriter.Create(relPath)                        
  29. if err != nil {                                                        
  30. fmt.Println(err)                                                   
  31. }                                                                      
  32. fsFile, err := os.Open(filePath)                                       
  33. if err != nil {                                                        
  34. fmt.Println(err)                                                   
  35. }                                                                      
  36. _, err = io.Copy(bundleFile, fsFile)                                   
  37. if err != nil {                                                        
  38. fmt.Println(err)                                                   
  39. }                                                                      
  40. }(filePath)                                                                
  41. }                                                                              
  42. wg.Wait()                                                                      
  43. err = bundleWriter.Close()                                                     
  44. if err != nil {                                                                
  45. return err                                                                 
  46. }                                                                              
  47. return nil                                                                     
  48. }

The error occurs here :

  1.         _, err = io.Copy(bundleFile, fsFile)                                   
  2. if err != nil {                                                        
  3. fmt.Println(err)                                                   
  4. }

The stack trace :

  1. goroutine 48 [running]:                                                            
  2. bufio.(*Writer).Write(0xc00002a100, {0xc00041a400?, 0x3d?, 0xc00041a400?})         
  3. /usr/local/go/src/bufio/bufio.go:670 +0x1c8                                    
  4. archive/zip.(*countWriter).Write(0xc00000c138, {0xc00041a400?, 0x3d?, 0x4afa20?})
  5. /usr/local/go/src/archive/zip/writer.go:601 +0x2e                              
  6. io.WriteString({0x4e7538, 0xc00000c138}, {0xc0000212c9, 0x3d})                     
  7. /usr/local/go/src/io/io.go:314 +0x91                                           
  8. archive/zip.writeHeader({0x4e7538, 0xc00000c138}, 0xc000220090)                    
  9. /usr/local/go/src/archive/zip/writer.go:422 +0x5ec                             
  10. archive/zip.(*Writer).CreateHeader(0xc0000760a0, 0xc00021e1b0)                     
  11. /usr/local/go/src/archive/zip/writer.go:378 +0x797                             
  12. archive/zip.(*Writer).Create(0x4e7698?, {0xc0000212c9, 0x3d})                      
  13. /usr/local/go/src/archive/zip/writer.go:223 +0x6c                              
  14. main.BuildArchive.func2({0xc0000212c0, 0x46})                                      
  15. /home/simba/go/src/foobar/main.go:79 +0x1c5                                    
  16. created by main.BuildArchive                                                       
  17. /home/simba/go/src/foobar/main.go:73 +0x5aa                                    
  18. panic: runtime error: slice bounds out of range [:4126] with capacity 4096         
  19. goroutine 6 [running]:                                                             
  20. bufio.(*Writer).Flush(0xc00002a100)                                                
  21. /usr/local/go/src/bufio/bufio.go:634 +0x171                                    
  22. bufio.(*Writer).Write(0xc00002a100, {0xc0001b4200?, 0xc000199b20?, 0xc000199b20?})
  23. /usr/local/go/src/bufio/bufio.go:672 +0xd8                                     
  24. archive/zip.(*countWriter).Write(0xc00000c138, {0xc0001b4200?, 0x0?, 0xc000199b40?})
  25. /usr/local/go/src/archive/zip/writer.go:601 +0x2e                              
  26. archive/zip.(*countWriter).Write(0xc000220018, {0xc0001b4200?, 0xc0001b02c0?, 0xc0001b02f0?})
  27. /usr/local/go/src/archive/zip/writer.go:601 +0x2e                              
  28. compress/flate.(*huffmanBitWriter).write(...)                                      
  29. /usr/local/go/src/compress/flate/huffman_bit_writer.go:136                     
  30. compress/flate.(*huffmanBitWriter).writeCode(0xc0001b41e0?, {0x6000?, 0x22?})   
  31. /usr/local/go/src/compress/flate/huffman_bit_writer.go:347 +0xe5               
  32. compress/flate.(*huffmanBitWriter).writeTokens(0xc0001b41e0, {0xc002558000, 0x4001, 0x403f800000403f?}, {0xc0001aa900, 0x11e, 0x108129000000000f?}, {0xc0001ac100, 0x1e, 0x1e})
  33. /usr/local/go/src/compress/flate/huffman_bit_writer.go:583 +0xb9               
  34. compress/flate.(*huffmanBitWriter).writeBlock(0xc0001b41e0, {0xc002558000?, 0x20?, 0xd79?}, 0x0, {0x0, 0x0, 0x0})
  35. /usr/local/go/src/compress/flate/huffman_bit_writer.go:495 +0x490              
  36. compress/flate.(*compressor).writeBlock(0xc0005a2000, {0xc002558000?, 0xc000032f00?, 0xc000199d28?}, 0x47739b?)
  37. /usr/local/go/src/compress/flate/deflate.go:170 +0x9c                          
  38. compress/flate.(*compressor).deflate(0xc0005a2000)                                 
  39. /usr/local/go/src/compress/flate/deflate.go:509 +0x59b                         
  40. compress/flate.(*compressor).write(0xc0005a2000, {0xc00256a000?, 0x8000, 0xf311b6fd?})
  41. /usr/local/go/src/compress/flate/deflate.go:554 +0x82                          
  42. compress/flate.(*Writer).Write(...)                                                
  43. /usr/local/go/src/compress/flate/deflate.go:712                                
  44. archive/zip.(*pooledFlateWriter).Write(0xc00020c040?, {0xc00256a000?, 0x8000?, 0x4af140?})
  45. /usr/local/go/src/archive/zip/register.go:51 +0xc5                             
  46. archive/zip.(*countWriter).Write(...)                                              
  47. /usr/local/go/src/archive/zip/writer.go:601                                    
  48. archive/zip.(*fileWriter).Write(0xc000222000, {0xc00256a000, 0x8000, 0x8000})   
  49. /usr/local/go/src/archive/zip/writer.go:533 +0x97                              
  50. io.copyBuffer({0x4e7558, 0xc000222000}, {0x4e7678, 0xc0001f8008}, {0x0, 0x0, 0x0})
  51. /usr/local/go/src/io/io.go:428 +0x204                                          
  52. io.Copy(...)                                                                       
  53. /usr/local/go/src/io/io.go:385                                                 
  54. main.BuildArchive.func2({0xc00001c0c0, 0x35})                                      
  55. /home/simba/go/src/foobar/main.go:89 +0x385                                    
  56. created by main.BuildArchive                                                       
  57. /home/simba/go/src/foobar/main.go:73 +0x5aa                                    
  58. exit status 2

答案1

得分: 3

zip.Writer 不支持并发使用。你启动了多个 goroutine，每个 goroutine 创建并写入 zip 条目（文件）。

Writer.Create() 的文档说明如下：

> Create 使用提供的名称将文件添加到 zip 文件中。它返回一个 Writer，用于写入文件内容。
>
> [...] 在下一次调用 Create、CreateHeader 或 Close 之前，必须将文件内容写入 io.Writer。

你不能并发地创建 zip 文件。每个 zip 条目的 io.Writer 都写入同一个底层文件（或者通常情况下写入同一个 io.Writer），即使你的应用程序没有崩溃或崩溃，生成的 zip 归档文件很可能是无效的。

英文:

zip.Writer is not safe for concurrent use. You launch multiple goroutines, each creating and writing to zip entries (files).

Writer.Create() documents that:

> Create adds a file to the zip file using the provided name. It returns a Writer to which the file contents should be written.
>
> [...] The file's contents must be written to the io.Writer before the next call to Create, CreateHeader, or Close.

You can't create a zip concurrently. io.Writers of each zip entry write to the same underlying file (or to the same io.Writer in general), even if your app doesn't panic or crash, the resulting zip archive will likely be invalid.

huangapple
  • 本文由 发表于 2022年11月28日 21:03:56
  • 转载请务必保留本文链接：https://go.coder-hub.com/74601005.html
  • go
  • goroutine
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定