Loop through all files in all folders recursively as fast as possible in GOLANG

huangapple
go评论117阅读模式
英文:

Loop through all files in all folders recursively as fast as possible in GOLANG

问题

我遇到了一个问题，即使我在论坛上花了一整天的时间，我仍然无法完全理解和解决它。

问题是这样的，我编写了一个函数，循环遍历所有的文件夹及其子文件夹，并包含了两个子函数：

  • 对于找到的每个文件，列出文件的名称。
  • 对于找到的每个文件夹，重新启动相同的父函数以再次查找子文件和子文件夹。

为了简化问题，该宏使用递归列出树中的所有文件。但我的目标是尽快完成，所以每当遇到一个新的文件夹时，我会运行一个新的goroutine。

问题是，当树结构太大时（文件夹和子文件夹中的文件太多），脚本会生成太多的线程，从而导致错误。所以我增加了这个限制，但突然间电脑不再工作了:/

所以我的问题是，如何创建一个适合我的代码的工作线程系统（带有池大小）？
无论我怎么看，我都不知道如何说，例如，生成新的goroutine直到达到某个限制，以便清空缓冲区。

源代码：
https://github.com/LaM0uette/FilesDIR/tree/V0.5

主要部分：

  1. package main
  3. import (
  4. 	"FilesDIR/globals"
  5. 	"FilesDIR/task"
  6. 	"fmt"
  7. 	"log"
  8. 	"runtime/debug"
  9. 	"sync"
  10. 	"time"
  11. )
  13. func main() {
  14. 	timeStart := time.Now()
  15. 	debug.SetMaxThreads(5 * 1000)
  17. 	var wg sync.WaitGroup
  19. 	err := task.LoopDirsFiles(globals.SrcPath, &wg) // globals.SrcPath = My path with ~2000000 files ( this is a serveur of my entreprise)
  20. 	if err != nil {
  21. 		log.Print(err.Error())
  22. 	}
  24. 	wg.Wait()
  26. 	fmt.Println("FINI: Nb Fichiers: ", task.Id)
  28. 	timeEnd := time.Since(timeStart)
  29. 	fmt.Println(timeEnd)
  30. }

任务部分：

  1. package task
  3. import (
  4. 	"fmt"
  5. 	"io/ioutil"
  6. 	"log"
  7. 	"os"
  8. 	"path/filepath"
  9. 	"strings"
  10. 	"sync"
  11. 	"time"
  12. )
  14. var Id = 0
  16. func LoopDirsFiles(path string, wg *sync.WaitGroup) error {
  17. 	wg.Add(1)
  18. 	defer wg.Done()
  20. 	files, err := ioutil.ReadDir(path)
  21. 	if err != nil {
  22. 		return err
  23. 	}
  25. 	for _, file := range files {
  26. 		if !file.IsDir() && !strings.Contains(file.Name(), "~") {
  27. 			fmt.Println(file.Name(), Id)
  28. 			Id++
  29. 		} else if file.IsDir() {
  30. 			go func() {
  31. 				err = LoopDirsFiles(filepath.Join(path, file.Name()), wg)
  32. 				if err != nil {
  33. 					log.Print(err)
  34. 				}
  35. 			}()
  36. 			time.Sleep(20 * time.Millisecond)
  37. 		}
  38. 	}
  39. 	return nil
  40. }

希望对你有帮助！

英文:

I'm facing a problem that even after spending the day on the forums I still can't quite understand and solve.

So here it is, I made a function that loops over all the folders as well as its sub-folders, and which has 2 sub-functions:
- For each file found, list the name of the file.
- For each folder found, restart the same parent function to find the child files and folders again.

To make it simpler, the macro lists all files in a tree with recursion. But my goal is to do it as fast as possible so I run a new goroutine every time I come across a new folder.

PROBLEM:
My problem is that when the tree structure is too large (too many folders in folders and subfolders...) the script generates too many threads and therefore gives me an error. So I increased this limit, but suddenly it's the pc that no longer wants :/

So my question is, how can I make a worker system (with poolsize) that fits my code?
No matter how much I look, I don't see how to say, for example, to generate new goroutines up to a certain limit, the time to empty the buffer.

Source code:
https://github.com/LaM0uette/FilesDIR/tree/V0.5

main:

  1. package main
  3. import (
  4. 	"FilesDIR/globals"
  5. 	"FilesDIR/task"
  6. 	"fmt"
  7. 	"log"
  8. 	"runtime/debug"
  9. 	"sync"
  10. 	"time"
  11. )
  13. func main() {
  14. 	timeStart := time.Now()
  15. 	debug.SetMaxThreads(5 * 1000)
  17. 	var wg sync.WaitGroup
  19. 	// task.DrawStart()
  21. 	/*
  22. 		err := task.LoopDir(globals.SrcPath)
  23. 		if err != nil {
  24. 			log.Print(err.Error())
  25. 		}
  26. 	*/
  28. 	err := task.LoopDirsFiles(globals.SrcPath, &wg) // globals.SrcPath = My path with ~2000000 files ( this is a serveur of my entreprise)
  29. 	if err != nil {
  30. 		log.Print(err.Error())
  31. 	}
  33. 	wg.Wait()
  35. 	fmt.Println("FINI: Nb Fichiers: ", task.Id)
  37. 	timeEnd := time.Since(timeStart)
  38. 	fmt.Println(timeEnd)
  39. }

task:

  1. package task
  3. import (
  4. 	"fmt"
  5. 	"io/ioutil"
  6. 	"log"
  7. 	"os"
  8. 	"path/filepath"
  9. 	"strings"
  10. 	"sync"
  11. 	"time"
  12. )
  14. var Id = 0
  16. // LoopDir TODO: Code à supprimer / Code to delete
  17. func LoopDir(path string) error {
  18. 	var wg sync.WaitGroup
  20. 	countDir := 0
  22. 	err := filepath.Walk(path, func(path string, info os.FileInfo, err error) error {
  23. 		if err != nil {
  24. 			return err
  25. 		}
  27. 		if info.IsDir() {
  28. 			wg.Add(1)
  29. 			countDir++
  31. 			go func() {
  32. 				err := loopFiles(path, &wg)
  33. 				if err != nil {
  34. 					log.Println(err.Error())
  35. 				}
  36. 			}()
  37. 		}
  39. 		return nil
  40. 	})
  41. 	if err != nil {
  42. 		return err
  43. 	}
  45. 	wg.Wait()
  46. 	fmt.Println("Finished", countDir, Id)
  47. 	return nil
  48. }
  50. // loopFiles TODO: Code à supprimer / Code to delete
  51. func loopFiles(path string, wg *sync.WaitGroup) error {
  53. 	files, err := ioutil.ReadDir(path)
  54. 	if err != nil {
  55. 		wg.Done()
  56. 		return err
  57. 	}
  59. 	for _, file := range files {
  60. 		if !file.IsDir() {
  61. 			go fmt.Println(file.Name())
  62. 			Id++
  63. 		}
  64. 	}
  66. 	wg.Done()
  67. 	return nil
  68. }
  70. func LoopDirsFiles(path string, wg *sync.WaitGroup) error {
  71. 	wg.Add(1)
  72. 	defer wg.Done()
  74. 	files, err := ioutil.ReadDir(path)
  75. 	if err != nil {
  76. 		return err
  77. 	}
  79. 	for _, file := range files {
  80. 		if !file.IsDir() && !strings.Contains(file.Name(), "~") {
  81. 			fmt.Println(file.Name(), Id)
  82. 			Id++
  83. 		} else if file.IsDir() {
  84. 			go func() {
  85. 				err = LoopDirsFiles(filepath.Join(path, file.Name()), wg)
  86. 				if err != nil {
  87. 					log.Print(err)
  88. 				}
  89. 			}()
  90. 			time.Sleep(20 * time.Millisecond)
  91. 		}
  92. 	}
  93. 	return nil
  94. }

答案1

得分: 2

如果您不想使用任何外部包，可以为文件处理创建一个单独的工作程序，然后启动所需数量的工作程序。之后，在主线程中递归进入树，并将作业发送给工作程序。如果任何工作程序“有时间”，它将从作业通道中获取下一个作业并处理它。

  1. var (
  2.     wg   *sync.WaitGroup
  3.     jobs chan string = make(chan string)
  4. )
  6. func loopFilesWorker() error {
  7.     for path := range jobs {
  8.         files, err := ioutil.ReadDir(path)
  9.         if err != nil {
  10.             wg.Done()
  11.             return err
  12.         }
  14.         for _, file := range files {
  15.             if !file.IsDir() {
  16.                 fmt.Println(file.Name())
  17.             }
  18.         }
  19.         wg.Done()
  20.     }
  21.     return nil
  22. }
  24. func LoopDirsFiles(path string) error {
  25.     files, err := ioutil.ReadDir(path)
  26.     if err != nil {
  27.         return err
  28.     }
  29.     //将此路径作为工作任务添加到工作程序中
  30.     //您必须在go例程中调用它，因为如果每个工作程序都忙碌，那么您必须等待通道空闲。
  31.     go func() {
  32.         wg.Add(1)
  33.         jobs <- path
  34.     }()
  35.     for _, file := range files {
  36.         if file.IsDir() {
  37.             //递归进入树
  38.             LoopDirsFiles(filepath.Join(path, file.Name()))
  39.         }
  40.     }
  41.     return nil
  42. }
  44. func main() {
  45.     //启动所需数量的工作程序，现在是10个工作程序
  46.     for w := 1; w <= 10; w++ {
  47.         go loopFilesWorker()
  48.     }
  49.     //开始递归
  50.     LoopDirsFiles(globals.SrcPath)
  51.     wg.Wait()
  52. }
英文:

If you don't want to use any external package, you can create a separate worker routine for file processing, then start as many workers you want. After that, go into the tree recursively in your main thread, and send out the jobs to the workers. If any worker "has time", it will pick up the following job from the jobs channel and process it.

  1. var (
  2. wg   *sync.WaitGroup
  3. jobs chan string = make(chan string)
  4. )
  5. func loopFilesWorker() error {
  6. for path := range jobs {
  7. files, err := ioutil.ReadDir(path)
  8. if err != nil {
  9. wg.Done()
  10. return err
  11. }
  12. for _, file := range files {
  13. if !file.IsDir() {
  14. fmt.Println(file.Name())
  15. }
  16. }
  17. wg.Done()
  18. }
  19. return nil
  20. }
  21. func LoopDirsFiles(path string) error {
  22. files, err := ioutil.ReadDir(path)
  23. if err != nil {
  24. return err
  25. }
  26. //Add this path as a job to the workers
  27. //You must call it in a go routine, since if every worker is busy, then you have to wait for the channel to be free.
  28. go func() {
  29. wg.Add(1)
  30. jobs &lt;- path
  31. }()
  32. for _, file := range files {
  33. if file.IsDir() {
  34. //Recursively go further in the tree
  35. LoopDirsFiles(filepath.Join(path, file.Name()))
  36. }
  37. }
  38. return nil
  39. }
  40. func main() {
  41. //Start as many workers you want, now 10 workers
  42. for w := 1; w &lt;= 10; w++ {
  43. go loopFilesWorker()
  44. }
  45. //Start the recursion
  46. LoopDirsFiles(globals.SrcPath)
  47. wg.Wait()
  48. }

huangapple
  • 本文由 发表于 2022年4月6日 20:32:49
  • 转载请务必保留本文链接：https://go.coder-hub.com/71766816.html
  • go
  • goroutine
  • loops
  • performance
  • threadpool
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定