Golang – why is string slice element not included in exec cat unless I sort it

huangapple go评论68阅读模式
英文:

Golang - why is string slice element not included in exec cat unless I sort it

问题

我在golang中遇到了一个有点奇怪的问题。基本上,我有一个字符串切片,表示文件路径。然后我运行一个cat命令来合并这些文件,然后进行排序、去重等操作。

以下是代码部分(其中'applicableReductions'是字符串切片):

    applicableReductions := []string{}

    for _, fqFromListName := range fqFromListNames {
         filePath := GetFilePath()
         //有问题的代码在这里
    }
    applicableReductions = append(applicableReductions, filePath)

    fileOut, err := os.Create(toListWriteTmpFilePath)
    if err != nil {
		return err
	}
    cat := exec.Command("cat", applicableReductions...)
	catStdOut, err := cat.StdoutPipe()
	if err != nil {
		return err
	}
	go func(cat *exec.Cmd) error {
		if err := cat.Start(); err != nil {
			return fmt.Errorf("文件合并错误(cat):%s", err)
		}
		return nil
	}(cat)

	// 初始化写入器并写入文件
	writer := bufio.NewWriter(fileOut)
	defer writer.Flush()

	_, err = io.Copy(writer, catStdOut)
	if err != nil {
		return err
	}

	if err = cat.Wait(); err != nil {
		return err
	}

	fDiff.StandardiseData(fileOut, toListUpdateFolderPath, list.Name)

上述代码运行良好。问题出现在我尝试将新元素附加到数组时。我有一个单独的函数,它从数据库内容创建一个新文件,然后将其添加到applicableReductions切片中。

func RetrieveDomainsFromDB(collection *Collection, listName, outputPath string) error {
	domains, err := domainReviews.GetDomainsForList(listName)
	if err != nil {
		return err
	}

	if len(domains) < 1 {
		return ErrNoDomainReviewsForList
	}

	fh, err := os.OpenFile(outputPath, os.O_RDWR, 0774)
	if err != nil {
		fh, err = os.Create(outputPath)
		if err != nil {
			return err
		}
	}
	defer fh.Close()
	_, err = fh.WriteString(strings.Join(domains, "\n"))
	if err != nil {
		return err
	}

	return nil
}

如果我调用上述函数并将filePath附加到applicableReduction切片中,它会出现在其中,但不会被cat命令调用。

为了澄清,当我将以下内容放在BROKE CODE GOES HERE处时:

	if dbSource {
        err = r.RetrieveDomainsFromDB(collection, ToListName, filePath)
	    if err != nil {
				return err
				continue
			}
      }

当执行fmt.Println(applicableReductions)时,可以看到filePath,但文件内容在cat输出文件中看不到。

我以为可能是文件写入的延迟,所以尝试添加了一个time.wait,但没有帮助。然而,我找到的解决办法是对切片进行排序,例如,在执行cat命令之前的这段代码解决了问题,但我不知道为什么:

sort.Strings(applicableReductions)

我确认所有文件都存在,无论是成功还是不成功的运行,唯一的区别是没有排序,最终附加文件的内容丢失了。

非常感谢Go专家的解释,如果需要更多信息或调试,我很乐意提供。

英文:

I have a slightly funky issue in golang. Essentially I have a slice of strings which represent file paths. I then run a cat against those filepaths to combine the files before sorting, deduping, etc.

here is the section of code (where 'applicableReductions' is the string slice):

    applicableReductions := []string{}

    for _, fqFromListName := range fqFromListNames {
         filePath := GetFilePath()
         //BROKE CODE GOES HERE
    }
    applicableReductions = append(applicableReductions, filePath)

    fileOut, err := os.Create(toListWriteTmpFilePath)
    if err != nil {
		return err
	}
    cat := exec.Command(&quot;cat&quot;, applicableReductions...)
	catStdOut, err := cat.StdoutPipe()
	if err != nil {
		return err
	}
	go func(cat *exec.Cmd) error {
		if err := cat.Start(); err != nil {
			return fmt.Errorf(&quot;File reduction error (cat) : %s&quot;, err)
		}
		return nil
	}(cat)

	// Init Writer &amp; write file
	writer := bufio.NewWriter(fileOut)
	defer writer.Flush()

	_, err = io.Copy(writer, catStdOut)
	if err != nil {
		return err
	}

	if err = cat.Wait(); err != nil {
		return err
	}

	fDiff.StandardiseData(fileOut, toListUpdateFolderPath, list.Name)

The above works fine. The problem comes when I try to append a new ele to the array. I have a seperate function which creates a new file from db content which is then added to the applicableReductions slice.

func RetrieveDomainsFromDB(collection *Collection, listName, outputPath string) error {
	domains, err := domainReviews.GetDomainsForList(listName)
	if err != nil {
		return err
	}

	if len(domains) &lt; 1 {
		return ErrNoDomainReviewsForList
	}

	fh, err := os.OpenFile(outputPath, os.O_RDWR, 0774)
	if err != nil {
		fh, err = os.Create(outputPath)
		if err != nil {
			return err
		}
	}
	defer fh.Close()
	_, err = fh.WriteString(strings.Join(domains, &quot;\n&quot;))
	if err != nil {
		return err
	}

	return nil
}

If I call the above function and append the filePath to the applicableReduction slice, it is in there but doesnt get called by cat.

To clarify, when I put the following where it says BROKE CODE GOES HERE:

	if dbSource {
        err = r.RetrieveDomainsFromDB(collection, ToListName, filePath)
	    if err != nil {
				return err
				continue
			}
      }

The filepath can be seen when doing fmt.Println(applicableReductions) but the content of the files contents are not seen in the cat output file.

I thought perhaps a delay in the file being written so i tried adding a time.wait, tis didnt help. However the solution I found was to sort the slice, e.g this code above the call to exec cat solves the problem but I dont know why:

sort.Strings(applicableReductions)

I have confirmed all files present on both successful and unsucessful runs the only difference is without the sort, the content of the final appended file is missing

An explanation from a go-pro out there would be very much appreciated, let me know if you need more info, debug - happy to oblige to understand

UPDATE
It has been suggested that this is the same issue as here: https://stackoverflow.com/questions/20195296/golang-append-an-item-to-a-slice, I think I understand the issue there and I'm not saying this isnt the same but I cannot see the same thing happenning - the slice in question is not touched from outside the main function (e.g. no editing of the slice in RetrieveDomainsFromDB function), I create the slice before a loop, append to it within a loop and then use it after the loop - Ive added an example at the top to show how the slice is built - please could someone clarify where this slice is being copied if this is the case

答案1

得分: 0

更新并关闭
请关闭问题 - 问题与字符串切片的使用无关。事实证明,在函数返回之前,我在最终输出文件中读取了bufio-writer尚未刷新的内容(在函数结束之前延迟刷新)。

我认为排序只是重新排列问题,所以我没有注意到问题仍然存在,或者可能给缓冲区一些时间来刷新。无论哪种方式,现在通过手动调用flush已经解决了。

感谢提供的所有帮助。

英文:

UPDATE AND CLOSE
Please close question - the issue was unrelated to the use of a string slice. Turns out that I was reading from the final output file before bufio-writer had been flushed (at end of function before defer flush kicked in on function return)

I think the sorting was just re-arranging the problem so I didnt notice it persisted or possibly giving some time for the buffer to flush. Either way sorted now with a manual call to flush.

Thanks for all help provided

huangapple
  • 本文由 发表于 2016年4月30日 01:01:32
  • 转载请务必保留本文链接:https://go.coder-hub.com/36943861.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定