频道挂起,可能没有在正确的位置关闭。

huangapple go评论122阅读模式
英文:

Channel hangs, probably not closing at the right place

问题

我正在尝试编写一个小程序来学习Go语言。该程序应该以尽可能高效和快速的方式递归解析一个路径,并输出包含完整文件名(包括路径)和文件的SHA256哈希值。

如果文件哈希生成失败,我希望保留错误并将其添加到字符串中(在哈希位置)。

结果应该在控制台上返回一个字符串,格式为:
文件名||哈希值

不幸的是,程序在某个点上停止运行。我猜想我的一些通道没有正确关闭,并且无限期地等待输入。我已经尝试了很长时间来解决这个问题,但没有成功。

有人知道为什么输出会停止吗?非常感谢,欢迎对Go语言新手提出任何建议和意见。

(我将不同的功能写成了单独的函数,因为在解决了这个问题之后,我想要添加其他功能。)

非常感谢!
Didier

以下是代码:

import (
	"crypto/sha256"
	"encoding/hex"
	"flag"
	"fmt"
	"io"
	"log"
	"os"
	"path/filepath"
	"time"
)

func main() {
	pathParam := flag.String("path", ".", "Enter Filesystem Path to list folders")
	flag.Parse()
	start := time.Now()
	run(*pathParam)
	elapsed := time.Since(start)
	log.Printf("Time elapsed: %v", elapsed)
}

func run(path string) {
	chashes := make(chan string, 50)
	cfiles := make(chan string)

	go func() {
		readfs(path, cfiles)
		defer close(cfiles)
	}()
	go func() {
		generateHash(cfiles, chashes)
	}()
	defer close(chashes)
	for hash := range chashes {
		fmt.Println(hash)
	}
}

func readfs(path string, cfiles chan string) {
	files, err := os.ReadDir(path)
	if err != nil {
		log.Fatalln(err)
	}
	for _, file := range files {
		filename := filepath.Join(path, file.Name())
		if file.IsDir() {
			readfs(filename, cfiles)
			continue
		} else {
			cfiles <- filename
		}
	}
}

func generateHash(cfiles chan string, chashes chan string) {
	for filename := range cfiles {
		go func(filename string) {
			var checksum string
			var oError bool = false
			file, err := os.Open(filename)
			if err != nil {
				oError = true
				errorMsg := "ERROR: " + err.Error()
				log.Println(errorMsg)
				checksum = errorMsg
			}
			defer file.Close()

			if !oError {
				hash := sha256.New()
				if _, err := io.Copy(hash, file); err != nil {
					errorMsg := "ERROR: " + err.Error()
					log.Println(errorMsg)
					checksum = errorMsg
				}
				if len(checksum) == 0 {
					checksum = hex.EncodeToString(hash.Sum(nil))
				}
			}
			chashes <- filename + "||" + checksum
		}(filename)
	} //for files
}
英文:

I'm trying to learn Go while writing a small program. The program should parse a PATH recursivelys as efficient and fast as possible and output the full filename (with the path included) and the sha256 file hash of the file.

If the file hashing generates fails, I wanna keep the error and add it to the string (at the hash position).

The result should return a string on the console like:
fileXYZ||hash

Unfortunately, the programs hangs at some point. I guess some of my channels are not closing properly and waiting indefinitely for input. I've been trying for quite some time to fix the problem, but without success.

Does anyone have an idea why the output hangs? Many many thx in advance, any input/advice for a Go newcomer is welcome too ;-).

(I wrote separate functions as I wanna add additional features after having fixed this issue.)

Thanks a lot!
Didier

Here is the code:

import (
&quot;crypto/sha256&quot;
&quot;encoding/hex&quot;
&quot;flag&quot;
&quot;fmt&quot;
&quot;io&quot;
&quot;log&quot;
&quot;os&quot;
&quot;path/filepath&quot;
&quot;time&quot;
)
func main() {
pathParam := flag.String(&quot;path&quot;, &quot;.&quot;, &quot;Enter Filesystem Path to list folders&quot;)
flag.Parse()
start := time.Now()
run(*pathParam)
elapsed := time.Since(start)
log.Printf(&quot;Time elapsed: %v&quot;, elapsed)
}
func run(path string) {
chashes := make(chan string, 50)
cfiles := make(chan string)
go func() {
readfs(path, cfiles)
defer close(cfiles)
}()
go func() {
generateHash(cfiles, chashes)
}()
defer close(chashes)
for hash := range chashes {
fmt.Println(hash)
}
}
func readfs(path string, cfiles chan string) {
files, err := os.ReadDir(path)
if err != nil {
log.Fatalln(err)
}
for _, file := range files {
filename := filepath.Join(path, file.Name())
if file.IsDir() {
readfs(filename, cfiles)
continue
} else {
cfiles &lt;- filename
}
}
}
func generateHash(cfiles chan string, chashes chan string) {
for filename := range cfiles {
go func(filename string) {
var checksum string
var oError bool = false
file, err := os.Open(filename)
if err != nil {
oError = true
errorMsg := &quot;ERROR: &quot; + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
defer file.Close()
if !oError {
hash := sha256.New()
if _, err := io.Copy(hash, file); err != nil {
errorMsg := &quot;ERROR: &quot; + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
if len(checksum) == 0 {
checksum = hex.EncodeToString(hash.Sum(nil))
}
}
chashes &lt;- filename + &quot;||&quot; + checksum
}(filename)
} //for files
}

答案1

得分: 0

以下循环陷入了死循环,因为chashes没有被关闭。

for hash := range chashes {
    fmt.Println(hash)
}

修复方法是在所有哈希计算完成后关闭chashes。使用sync.WaitGroup等待哈希计算完成。

func generateHash(cfiles chan string, chashes chan string) {
    var wg sync.WaitGroup
    for filename := range cfiles {
        wg.Add(1)
        go func(filename string) {
            defer wg.Done()
            var checksum string
            var oError bool = false
            file, err := os.Open(filename)
            if err != nil {
                oError = true
                errorMsg := "ERROR: " + err.Error()
                log.Println(errorMsg)
                checksum = errorMsg
            }
            defer file.Close()

            if !oError {
                hash := sha256.New()
                if _, err := io.Copy(hash, file); err != nil {
                    errorMsg := "ERROR: " + err.Error()
                    log.Println(errorMsg)
                    checksum = errorMsg
                }
                if len(checksum) == 0 {
                    checksum = hex.EncodeToString(hash.Sum(nil))
                }
            }
            chashes <- filename + "||" + checksum
        }(filename)
    } //for files

    // 等待哈希计算完成
    wg.Wait()

    // 关闭通道以使主函数中的 for range 在 chashes 上退出
    close(chashes)
}

run()函数中移除defer close(chashes)

在Go Playground上运行示例

英文:

The following loop hangs because chashes is not closed.

for hash := range chashes {
fmt.Println(hash)
}

Fix by closing chashes after all the hashers are completed. Use a sync.WaitGroup to wait for the hashers to complete.

func generateHash(cfiles chan string, chashes chan string) {
var wg sync.WaitGroup
for filename := range cfiles {
wg.Add(1)
go func(filename string) {
defer wg.Done()
var checksum string
var oError bool = false
file, err := os.Open(filename)
if err != nil {
oError = true
errorMsg := &quot;ERROR: &quot; + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
defer file.Close()
if !oError {
hash := sha256.New()
if _, err := io.Copy(hash, file); err != nil {
errorMsg := &quot;ERROR: &quot; + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
if len(checksum) == 0 {
checksum = hex.EncodeToString(hash.Sum(nil))
}
}
chashes &lt;- filename + &quot;||&quot; + checksum
}(filename)
} //for files
// Wait for the hashers to complete.
wg.Wait()
// Close the channel to cause main() to break
// out of for range on chashes.
close(chashes)
}

Remove defer close(chashes) from run().

Run an example on the Go playground.

huangapple
  • 本文由 发表于 2022年1月30日 03:20:10
  • 转载请务必保留本文链接:https://go.coder-hub.com/70908948.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定