How to perform file system scanning

1. I need to write a function which when given the path of a folder
   scans the files rooted at that folder.
2. And then I need to display the directory structure at that folder.

I know how to do 2 (I am going to use jstree to display it in the browser).

EDIT FOR 1.16: Enough people still hit this answer, that I thought I'd update it for Go 1.16.

The function filepath.WalkDir introduced in Go 1.16 has better performance than filepath.Walk mentioned in the previous edit. Here's a working example:

package main

import (
	"flag"
	"fmt"
	"io/fs"
	"path/filepath"
)

func visit(path string, di fs.DirEntry, err error) error {
	fmt.Printf("Visited: %s\n", path)
	return nil
}

func main() {
	flag.Parse()
	root := flag.Arg(0)
	err := filepath.WalkDir(root, visit)
	fmt.Printf("filepath.WalkDir() returned %v\n", err)
}

EDIT: Enough people still hit this answer, that I thought I'd update it for the Go1 API. This is a working example of filepath.Walk(). The original is below.

package main

import (
  "path/filepath"
  "os"
  "flag"
  "fmt"
)

func visit(path string, f os.FileInfo, err error) error {
  fmt.Printf("Visited: %s\n", path)
  return nil
} 


func main() {
  flag.Parse()
  root := flag.Arg(0)
  err := filepath.Walk(root, visit)
  fmt.Printf("filepath.Walk() returned %v\n", err)
}

Please note that filepath.Walk walks the directory tree recursively.

This is an example run:

$ mkdir -p dir1/dir2
$ touch dir1/file1 dir1/dir2/file2
$ go run walk.go dir1
Visited: dir1
Visited: dir1/dir2
Visited: dir1/dir2/file2
Visited: dir1/file1
filepath.Walk() returned <nil>

ORIGINAL ANSWER FOLLOWS: The interface for walking file paths has changed as of weekly.2011-09-16, see http://groups.google.com/group/golang-nuts/msg/e304dd9cf196a218. The code below will not work for release versions of GO in the near future.

There's actually a function in the standard lib just for this: filepath.Walk.

package main

import (
	"path/filepath"
	"os"
	"flag"
)

type visitor int

// THIS CODE NO LONGER WORKS, PLEASE SEE ABOVE
func (v visitor) VisitDir(path string, f *os.FileInfo) bool {
	println(path)
	return true
} 

func (v visitor) VisitFile(path string, f *os.FileInfo) {
	println(path)
}

func main() {
	root := flag.Arg(0)
	filepath.Walk(root, visitor(0), nil)
}

Here's a way to obtain file information for the files in a directory.

package main

import (
	"fmt"
	"os"
	"path/filepath"
)

func main() {
	dirname := "." + string(filepath.Separator)
	d, err := os.Open(dirname)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	defer d.Close()
	fi, err := d.Readdir(-1)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	for _, fi := range fi {
		if fi.Mode().IsRegular() {
			fmt.Println(fi.Name(), fi.Size(), "bytes")
		}
	}
}

Here is an example to loop through all files and directories recursively. Note that if you want to know whether the path you're appending is a directory just check "f.IsDir()".

package main

import (
	"fmt"
	"os"
	"path/filepath"
)

func main() {
	searchDir := "c:/path/to/dir"
	
	fileList := []string{}
	err := filepath.Walk(searchDir, func(path string, f os.FileInfo, err error) error {
		fileList = append(fileList, path)
		return nil
	})

	for _, file := range fileList {
		fmt.Println(file)
	}
}

Package github.com/kr/fs provides a Walker with a very interesting API.

Go standard package ioutil has built in function for this case scenario see below example

func searchFiles(dir string) { // dir is the parent directory you what to search
	files, err := ioutil.ReadDir(dir)
	if err != nil {
		log.Fatal(err)
	}

	for _, file := range files {
		fmt.Println(file.Name())
	}
}

Note that "Walk does not follow symbolic links" so if you are looking to write a function that does that I recommend ioutil.ReadDir. My own benchmark test showed that it is faster and less memory intensive than filepath.Glob.

Additionally, ioutil.ReadDir is sorting files by basename using basic string comparison (strA > strB). As a devops guy, I generally sort dir names by doing a reverse numerical comparison (latest build first for example). If that is also your case then it is better to call os.ReadDir directly (ioutil.ReadDir is calling this under the covers) and do the sorting yourself.

Here is an example of the ReadDir part with Numerical sort:

// ReadDirNumSort - Same as ioutil/ReadDir but uses returns a Numerically
// Sorted file list.
//
// Taken from https://golang.org/src/io/ioutil/ioutil.go
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//
// Modified Sort method to use Numerically sorted names instead.
// It also allows reverse sorting.
func ReadDirNumSort(dirname string, reverse bool) ([]os.FileInfo, error) {
	f, err := os.Open(dirname)
	if err != nil {
		return nil, err
	}
	list, err := f.Readdir(-1)
	f.Close()
	if err != nil {
		return nil, err
	}
	if reverse {
		sort.Sort(sort.Reverse(byName(list)))
	} else {
		sort.Sort(byName(list))
	}
	return list, nil
}

// byName implements sort.Interface.
type byName []os.FileInfo

func (f byName) Len() int      { return len(f) }
func (f byName) Swap(i, j int) { f[i], f[j] = f[j], f[i] }
func (f byName) Less(i, j int) bool {
	nai, err := strconv.Atoi(f[i].Name())
	if err != nil {
		return f[i].Name() < f[j].Name()
	}
	naj, err := strconv.Atoi(f[j].Name())
	if err != nil {
		return f[i].Name() < f[j].Name()
	}
	return nai < naj
}

You might want to do function currying here, so that you are able to fully utilise the search

func visit(files *[]string) filepath.WalkFunc {
    return func (path string, info os.FileInfo, err error) error {
               // maybe do this in some if block
               *files = append(*files, path)
               return nil
           }
}

func printAllFilesRecursively(input string) {
	filesInfo, _ := ioutil.ReadDir(input)
	for _, each := range filesInfo {
		println(input + "/" + each.Name())
		if each.IsDir() {
			printAllFilesRecursively(input + "/" + each.Name())
		}
	}
}