golang os *File.Readdir在所有文件上使用lstat。它可以进行优化吗？

huangapple
go评论110阅读模式
英文:

golang os *File.Readdir using lstat on all files. Can it be optimised?

问题

我正在编写一个程序，使用os.File.Readdir从父目录中找到包含大量文件的所有子目录，但是运行strace以查看系统调用的计数时，发现Go版本在父目录中的所有文件/目录上都使用了lstat()。（我现在正在使用/usr/bin目录进行测试）

Go代码：

  1. package main
  2. import (
  3.     "fmt"
  4.     "os"
  5. )
  6. func main() {
  7.     x, err := os.Open("/usr/bin")
  8.     if err != nil {
  9.         panic(err)
  10.     }
  11.     y, err := x.Readdir(0)
  12.     if err != nil {
  13.         panic(err)
  14.     }
  15.     for _, i := range y {
  16.         fmt.Println(i)
  17.     }
  18. }

程序的strace（不包括跟踪线程）：

  1. % time     seconds  usecs/call     calls    errors syscall
  2. ------ ----------- ----------- --------- --------- ----------------
  3.  93.62    0.004110           2      2466           write
  4.   3.46    0.000152           7        22           getdents64
  5.   2.92    0.000128           0      2466           lstat // 这个随着文件数量的增加而增加。
  6.   0.00    0.000000           0        11           mmap
  7.   0.00    0.000000           0         1           munmap
  8.   0.00    0.000000           0       114           rt_sigaction
  9.   0.00    0.000000           0         8           rt_sigprocmask
  10.   0.00    0.000000           0         1           sched_yield
  11.   0.00    0.000000           0         3           clone
  12.   0.00    0.000000           0         1           execve
  13.   0.00    0.000000           0         2           sigaltstack
  14.   0.00    0.000000           0         1           arch_prctl
  15.   0.00    0.000000           0         1           gettid
  16.   0.00    0.000000           0        57           futex
  17.   0.00    0.000000           0         1           sched_getaffinity
  18.   0.00    0.000000           0         1           openat
  19. ------ ----------- ----------- --------- --------- ----------------
  20. 100.00    0.004390                  5156           total

我用C的readdir()进行了相同的测试，但没有看到这种行为。

C代码：

  1. #include <stdio.h>
  2. #include <dirent.h>
  4. int main (void) {
  5.     DIR* dir_p;
  6.     struct dirent* dir_ent;
  8.     dir_p = opendir ("/usr/bin");
  10.     if (dir_p != NULL) {
  11.         while ((dir_ent = readdir (dir_p)) != NULL) {
  12.             if (dir_ent->d_type == DT_DIR) {
  13.                 printf("%s is a directory\n", dir_ent->d_name);
  14.             } else {
  15.                 printf("%s is not a directory\n", dir_ent->d_name);
  16.             }
  17.             printf("\n");
  18.         }
  19.         (void) closedir(dir_p);
  20.     }
  21.     else
  22.         perror ("Couldn't open the directory");
  24.     return 0;
  25. }

程序的strace：

  1. % time     seconds  usecs/call     calls    errors syscall
  2. ------ ----------- ----------- --------- --------- ----------------
  3. 100.00    0.000128           0      2468           write
  4.   0.00    0.000000           0         1           read
  5.   0.00    0.000000           0         3           open
  6.   0.00    0.000000           0         3           close
  7.   0.00    0.000000           0         4           fstat
  8.   0.00    0.000000           0         8           mmap
  9.   0.00    0.000000           0         3           mprotect
  10.   0.00    0.000000           0         1           munmap
  11.   0.00    0.000000           0         3           brk
  12.   0.00    0.000000           0         3         3 access
  13.   0.00    0.000000           0         1           execve
  14.   0.00    0.000000           0         4           getdents
  15.   0.00    0.000000           0         1           arch_prctl
  16. ------ ----------- ----------- --------- --------- ----------------
  17. 100.00    0.000128                  2503         3 total

我知道dirent结构中POSIX.1规定的唯一字段是d_name和d_ino，但我正在为特定的文件系统编写此程序。

我尝试了*File.Readdirnames()，它不使用lstat并返回所有文件和目录的列表，但是要确定返回的字符串是文件还是目录，最终还是需要进行lstat。

  • 我想知道是否有可能以避免不必要地对所有文件执行lstat()的方式重新编写go程序。我注意到C程序使用了以下系统调用：open("/usr/bin", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFDIR|0755, st_size=69632, ...}) = 0 brk(NULL) = 0x1098000 brk(0x10c1000) = 0x10c1000 getdents(3, /* 986 entries */, 32768) = 32752
  • 这是否属于过早优化的范畴，我不应该担心？我提出这个问题是因为要监视的目录中的文件数量将会非常庞大，而且C和GO版本之间的系统调用差异几乎是两倍，这将会对磁盘造成影响。
英文:

I am writing a program that finds all sub-directories from a parent directory which contains a huge number of files using os.File.Readdir, but running an strace to see the count of systemcalls showed that the go version is using an lstat() on all the files/directories present in the parent directory. (I am testing this with /usr/bin directory for now)

Go code:

  1. package main
  2. import (
  3.     	&quot;fmt&quot;
  4. 	&quot;os&quot;
  5. )
  6. func main() {
  7. 	x, err := os.Open(&quot;/usr/bin&quot;)
  8. 	if err != nil {
  9. 		panic(err)
  10. 	}
  11. 	y, err := x.Readdir(0)
  12. 	if err != nil {
  13. 		panic(err)
  14. 	}
  15. 	for _, i := range y {
  16. 	fmt.Println(i)
  17. 	}
  19. }

Strace on the program (without following threads):

  1. % time     seconds  usecs/call     calls    errors syscall
  2. ------ ----------- ----------- --------- --------- ----------------
  3.  93.62    0.004110           2      2466           write
  4.   3.46    0.000152           7        22           getdents64
  5.   2.92    0.000128           0      2466           lstat // this increases with increase in no. of files.
  6.   0.00    0.000000           0        11           mmap
  7.   0.00    0.000000           0         1           munmap
  8.   0.00    0.000000           0       114           rt_sigaction
  9.   0.00    0.000000           0         8           rt_sigprocmask
  10.   0.00    0.000000           0         1           sched_yield
  11.   0.00    0.000000           0         3           clone
  12.   0.00    0.000000           0         1           execve
  13.   0.00    0.000000           0         2           sigaltstack
  14.   0.00    0.000000           0         1           arch_prctl
  15.   0.00    0.000000           0         1           gettid
  16.   0.00    0.000000           0        57           futex
  17.   0.00    0.000000           0         1           sched_getaffinity
  18.   0.00    0.000000           0         1           openat
  19. ------ ----------- ----------- --------- --------- ----------------
  20. 100.00    0.004390                  5156           total

I tested the same with the C's readdir() without seeing this behaviour.

C code:

  1. #include &lt;stdio.h&gt;
  2. #include &lt;dirent.h&gt;
  4. int main (void) {
  5.     DIR* dir_p;
  6.     struct dirent* dir_ent;
  8.     dir_p = opendir (&quot;/usr/bin&quot;);
  10.     if (dir_p != NULL) {
  11.         // The readdir() function returns a pointer to a dirent structure representing the next
  12.         // directory entry in the directory stream pointed to by dirp.
  13.         // It returns NULL on reaching the end of the directory stream or if an error occurred.
  14.         while ((dir_ent = readdir (dir_p)) != NULL) {
  15.             // printf(&quot;%s&quot;, dir_ent-&gt;d_name);
  16.             // printf(&quot;%d&quot;, dir_ent-&gt;d_type);
  17.             if (dir_ent-&gt;d_type == DT_DIR) {
  18.                 printf(&quot;%s is a directory&quot;, dir_ent-&gt;d_name);
  19.             } else {
  20.                 printf(&quot;%s is not a directory&quot;, dir_ent-&gt;d_name);
  21.             }
  23.             printf(&quot;\n&quot;);
  24.         }
  25.             (void) closedir(dir_p);
  27.     }
  28.     else
  29.         perror (&quot;Couldn&#39;t open the directory&quot;);
  31.     return 0;
  32. }

Strace on the program:

  1. % time     seconds  usecs/call     calls    errors syscall
  2. ------ ----------- ----------- --------- --------- ----------------
  3. 100.00    0.000128           0      2468           write
  4.   0.00    0.000000           0         1           read
  5.   0.00    0.000000           0         3           open
  6.   0.00    0.000000           0         3           close
  7.   0.00    0.000000           0         4           fstat
  8.   0.00    0.000000           0         8           mmap
  9.   0.00    0.000000           0         3           mprotect
  10.   0.00    0.000000           0         1           munmap
  11.   0.00    0.000000           0         3           brk
  12.   0.00    0.000000           0         3         3 access
  13.   0.00    0.000000           0         1           execve
  14.   0.00    0.000000           0         4           getdents
  15.   0.00    0.000000           0         1           arch_prctl
  16. ------ ----------- ----------- --------- --------- ----------------
  17. 100.00    0.000128                  2503         3 total

I am aware that the only fields in the dirent structure that are mandated by POSIX.1 are d_name and d_ino, but I am writing this for a specific filesystem.

Tried *File.Readdirnames(), which doesn't use an lstat and gives a list of all files and directories, but to see if the returned string is a file or a directory will eventually do an lstat again.

  • I was wondering if it is possible to re-write the go program in a way to avoid the lstat() on all the files un-necessarily. I could see the C program is using the following syscalls. open(&quot;/usr/bin&quot;, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
    fstat(3, {st_mode=S_IFDIR|0755, st_size=69632, ...}) = 0
    brk(NULL) = 0x1098000
    brk(0x10c1000) = 0x10c1000
    getdents(3, /* 986 entries */, 32768) = 32752
  • Is this something like a premature optimisation, which I shouldn't be worried about? I raised this question because the number of files in the directory being monitored will be having huge number of small archived files, and the difference in systemcalls is almost twice between C and GO version, which will be hitting the disk.

答案1

得分: 7

dirent包看起来可以实现你所需要的功能。以下是用Go语言编写的C示例：

  1. package main
  3. import (
  4. 	"bytes"
  5. 	"fmt"
  6. 	"io"
  8. 	"github.com/EricLagergren/go-gnulib/dirent"
  9. 	"golang.org/x/sys/unix"
  10. )
  12. func int8ToString(s []int8) string {
  13. 	var buff bytes.Buffer
  14. 	for _, chr := range s {
  15. 		if chr == 0x00 {
  16. 			break
  17. 		}
  18. 		buff.WriteByte(byte(chr))
  19. 	}
  20. 	return buff.String()
  21. }
  23. func main() {
  24. 	stream, err := dirent.Open("/usr/bin")
  25. 	if err != nil {
  26. 		panic(err)
  27. 	}
  28. 	defer stream.Close()
  29. 	for {
  30. 		entry, err := stream.Read()
  31. 		if err != nil {
  32. 			if err == io.EOF {
  33. 				break
  34. 			}
  35. 			panic(err)
  36. 		}
  38. 		name := int8ToString(entry.Name[:])
  39. 		if entry.Type == unix.DT_DIR {
  40. 			fmt.Printf("%s 是一个目录\n", name)
  41. 		} else {
  42. 			fmt.Printf("%s 不是一个目录\n", name)
  43. 		}
  44. 	}
  45. }
英文:

The package dirent looks like it accomplishes what you are looking for. Below is your C example written in Go:

  1. package main
  3. import (
  4. 	&quot;bytes&quot;
  5. 	&quot;fmt&quot;
  6. 	&quot;io&quot;
  8. 	&quot;github.com/EricLagergren/go-gnulib/dirent&quot;
  9. 	&quot;golang.org/x/sys/unix&quot;
  10. )
  12. func int8ToString(s []int8) string {
  13. 	var buff bytes.Buffer
  14. 	for _, chr := range s {
  15. 		if chr == 0x00 {
  16. 			break
  17. 		}
  18. 		buff.WriteByte(byte(chr))
  19. 	}
  20. 	return buff.String()
  21. }
  23. func main() {
  24. 	stream, err := dirent.Open(&quot;/usr/bin&quot;)
  25. 	if err != nil {
  26. 		panic(err)
  27. 	}
  28. 	defer stream.Close()
  29. 	for {
  30. 		entry, err := stream.Read()
  31. 		if err != nil {
  32. 			if err == io.EOF {
  33. 				break
  34. 			}
  35. 			panic(err)
  36. 		}
  38. 		name := int8ToString(entry.Name[:])
  39. 		if entry.Type == unix.DT_DIR {
  40. 			fmt.Printf(&quot;%s is a directory\n&quot;, name)
  41. 		} else {
  42. 			fmt.Printf(&quot;%s is not a directory\n&quot;, name)
  43. 		}
  44. 	}
  45. }

答案2

得分: 2

从Go 1.16（2021年2月）开始，一个很好的选择是使用os.ReadDir：

  1. package main
  2. import "os"
  4. func main() {
  5.    files, e := os.ReadDir(".")
  6.    if e != nil {
  7.       panic(e)
  8.    }
  9.    for _, file := range files {
  10.       println(file.Name())
  11.    }
  12. }

os.ReadDir返回的是fs.DirEntry而不是fs.FileInfo，这意味着Size和ModTime方法被省略了，使得过程更加高效。

https://golang.org/pkg/os#ReadDir

英文:

Starting with Go 1.16 (Feb 2021), a good option is os.ReadDir:

  1. package main
  2. import &quot;os&quot;
  4. func main() {
  5.    files, e := os.ReadDir(&quot;.&quot;)
  6.    if e != nil {
  7.       panic(e)
  8.    }
  9.    for _, file := range files {
  10.       println(file.Name())
  11.    }
  12. }

os.ReadDir returns fs.DirEntry instead of fs.FileInfo, which means that
Size and ModTime methods are omitted, making the process more efficient.

https://golang.org/pkg/os#ReadDir

huangapple
  • 本文由 发表于 2017年1月2日 05:07:33
  • 转载请务必保留本文链接：https://go.coder-hub.com/41419056.html
  • c
  • filesystems
  • go
  • linux
  • stat
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定