英文:
Channel hangs, probably not closing at the right place
问题
我正在尝试编写一个小程序来学习Go语言。该程序应该以尽可能高效和快速的方式递归解析一个路径,并输出包含完整文件名(包括路径)和文件的SHA256哈希值。
如果文件哈希生成失败,我希望保留错误并将其添加到字符串中(在哈希位置)。
结果应该在控制台上返回一个字符串,格式为:
文件名||哈希值
不幸的是,程序在某个点上停止运行。我猜想我的一些通道没有正确关闭,并且无限期地等待输入。我已经尝试了很长时间来解决这个问题,但没有成功。
有人知道为什么输出会停止吗?非常感谢,欢迎对Go语言新手提出任何建议和意见。
(我将不同的功能写成了单独的函数,因为在解决了这个问题之后,我想要添加其他功能。)
非常感谢!
Didier
以下是代码:
import (
"crypto/sha256"
"encoding/hex"
"flag"
"fmt"
"io"
"log"
"os"
"path/filepath"
"time"
)
func main() {
pathParam := flag.String("path", ".", "Enter Filesystem Path to list folders")
flag.Parse()
start := time.Now()
run(*pathParam)
elapsed := time.Since(start)
log.Printf("Time elapsed: %v", elapsed)
}
func run(path string) {
chashes := make(chan string, 50)
cfiles := make(chan string)
go func() {
readfs(path, cfiles)
defer close(cfiles)
}()
go func() {
generateHash(cfiles, chashes)
}()
defer close(chashes)
for hash := range chashes {
fmt.Println(hash)
}
}
func readfs(path string, cfiles chan string) {
files, err := os.ReadDir(path)
if err != nil {
log.Fatalln(err)
}
for _, file := range files {
filename := filepath.Join(path, file.Name())
if file.IsDir() {
readfs(filename, cfiles)
continue
} else {
cfiles <- filename
}
}
}
func generateHash(cfiles chan string, chashes chan string) {
for filename := range cfiles {
go func(filename string) {
var checksum string
var oError bool = false
file, err := os.Open(filename)
if err != nil {
oError = true
errorMsg := "ERROR: " + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
defer file.Close()
if !oError {
hash := sha256.New()
if _, err := io.Copy(hash, file); err != nil {
errorMsg := "ERROR: " + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
if len(checksum) == 0 {
checksum = hex.EncodeToString(hash.Sum(nil))
}
}
chashes <- filename + "||" + checksum
}(filename)
} //for files
}
英文:
I'm trying to learn Go while writing a small program. The program should parse a PATH recursivelys as efficient and fast as possible and output the full filename (with the path included) and the sha256 file hash of the file.
If the file hashing generates fails, I wanna keep the error and add it to the string (at the hash position).
The result should return a string on the console like:
fileXYZ||hash
Unfortunately, the programs hangs at some point. I guess some of my channels are not closing properly and waiting indefinitely for input. I've been trying for quite some time to fix the problem, but without success.
Does anyone have an idea why the output hangs? Many many thx in advance, any input/advice for a Go newcomer is welcome too ;-).
(I wrote separate functions as I wanna add additional features after having fixed this issue.)
Thanks a lot!
Didier
Here is the code:
import (
"crypto/sha256"
"encoding/hex"
"flag"
"fmt"
"io"
"log"
"os"
"path/filepath"
"time"
)
func main() {
pathParam := flag.String("path", ".", "Enter Filesystem Path to list folders")
flag.Parse()
start := time.Now()
run(*pathParam)
elapsed := time.Since(start)
log.Printf("Time elapsed: %v", elapsed)
}
func run(path string) {
chashes := make(chan string, 50)
cfiles := make(chan string)
go func() {
readfs(path, cfiles)
defer close(cfiles)
}()
go func() {
generateHash(cfiles, chashes)
}()
defer close(chashes)
for hash := range chashes {
fmt.Println(hash)
}
}
func readfs(path string, cfiles chan string) {
files, err := os.ReadDir(path)
if err != nil {
log.Fatalln(err)
}
for _, file := range files {
filename := filepath.Join(path, file.Name())
if file.IsDir() {
readfs(filename, cfiles)
continue
} else {
cfiles <- filename
}
}
}
func generateHash(cfiles chan string, chashes chan string) {
for filename := range cfiles {
go func(filename string) {
var checksum string
var oError bool = false
file, err := os.Open(filename)
if err != nil {
oError = true
errorMsg := "ERROR: " + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
defer file.Close()
if !oError {
hash := sha256.New()
if _, err := io.Copy(hash, file); err != nil {
errorMsg := "ERROR: " + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
if len(checksum) == 0 {
checksum = hex.EncodeToString(hash.Sum(nil))
}
}
chashes <- filename + "||" + checksum
}(filename)
} //for files
}
答案1
得分: 0
以下循环陷入了死循环,因为chashes
没有被关闭。
for hash := range chashes {
fmt.Println(hash)
}
修复方法是在所有哈希计算完成后关闭chashes
。使用sync.WaitGroup
等待哈希计算完成。
func generateHash(cfiles chan string, chashes chan string) {
var wg sync.WaitGroup
for filename := range cfiles {
wg.Add(1)
go func(filename string) {
defer wg.Done()
var checksum string
var oError bool = false
file, err := os.Open(filename)
if err != nil {
oError = true
errorMsg := "ERROR: " + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
defer file.Close()
if !oError {
hash := sha256.New()
if _, err := io.Copy(hash, file); err != nil {
errorMsg := "ERROR: " + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
if len(checksum) == 0 {
checksum = hex.EncodeToString(hash.Sum(nil))
}
}
chashes <- filename + "||" + checksum
}(filename)
} //for files
// 等待哈希计算完成
wg.Wait()
// 关闭通道以使主函数中的 for range 在 chashes 上退出
close(chashes)
}
从run()
函数中移除defer close(chashes)
。
英文:
The following loop hangs because chashes
is not closed.
for hash := range chashes {
fmt.Println(hash)
}
Fix by closing chashes
after all the hashers are completed. Use a sync.WaitGroup to wait for the hashers to complete.
func generateHash(cfiles chan string, chashes chan string) {
var wg sync.WaitGroup
for filename := range cfiles {
wg.Add(1)
go func(filename string) {
defer wg.Done()
var checksum string
var oError bool = false
file, err := os.Open(filename)
if err != nil {
oError = true
errorMsg := "ERROR: " + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
defer file.Close()
if !oError {
hash := sha256.New()
if _, err := io.Copy(hash, file); err != nil {
errorMsg := "ERROR: " + err.Error()
log.Println(errorMsg)
checksum = errorMsg
}
if len(checksum) == 0 {
checksum = hex.EncodeToString(hash.Sum(nil))
}
}
chashes <- filename + "||" + checksum
}(filename)
} //for files
// Wait for the hashers to complete.
wg.Wait()
// Close the channel to cause main() to break
// out of for range on chashes.
close(chashes)
}
Remove defer close(chashes)
from run()
.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论