2022年7月29日 17:59:53go评论167阅读模式

英文:

Why are IO operations slower in Goroutines?

问题

我正在从goroutine中下载图像后处理IO操作。

在测试过程中出现了一个问题。

在goroutine中下载图像后，我发现IO操作非常缓慢。

相反，在goroutine之外进行IO操作的速度更快。

我想知道为什么会这样？

以下是测试的源代码。

type ImageResult struct {
	TargetImagePath string
	Success         bool
}

type ImageDownloadResult struct {
	TargetImagePath string
	Response        *http.Response
	Success         bool
}

func main() {
	downloadUrls := []string{
		"https://myhost.com/item/image/path/name_01.png",
		"https://myhost.com/item/image/path/name_02.png",
		"https://myhost.com/item/image/path/name_03.png",
		"https://myhost.com/item/image/path/name_05.png",
		"https://myhost.com/item/image/path/name_07.png",
		"https://myhost.com/item/image/path/name_08.png",
		"https://myhost.com/item/image/path/name_09.png",
		"https://myhost.com/item/image/path/name_10.png",
		"https://myhost.com/item/image/path/name_11.png",
		"https://myhost.com/item/image/path/name_12.png",
		"https://myhost.com/item/image/path/name_13.png",
		"https://myhost.com/item/image/path/name_14.png",
		"https://myhost.com/item/image/path/name_16.png",
	}

	startAsyncIO := time.Now()
	resultChannel := make(chan ImageResult)
	for _, downloadUrl := range downloadUrls {
		go HttpFileDownLoadAndIOWithAsync(downloadUrl, resultChannel)
	}

	for i := 0; i < len(downloadUrls); i++ {
		<-resultChannel
	}
	fmt.Println("Total Time Async IO: ", time.Now().Sub(startAsyncIO))

	fmt.Println("=======================VS========================")

	startSyncIO := time.Now()
	resultChannel2 := make(chan ImageDownloadResult)
	for _, downloadUrl := range downloadUrls {
		go HttpFileDownLoadWithAsync(downloadUrl, resultChannel2)

	}

	for i := 0; i < len(downloadUrls); i++ {
		result := <-resultChannel2
		getBytesFromDownloadData(result.Response, result.TargetImagePath)
	}
	fmt.Println("Total Time Sync IO: ", time.Now().Sub(startSyncIO))
}

func HttpFileDownLoadAndIOWithAsync(downloadUrl string, imageResult chan ImageResult) {
	result := ImageResult{
		TargetImagePath: downloadUrl,
	}

	client := http.Client{
		CheckRedirect: func(r *http.Request, via []*http.Request) error {
			r.URL.Opaque = r.URL.Path
			return nil
		},
	}
	// Put content on file
	downStart := time.Now()
	resp, _ := client.Get(downloadUrl)
	downEnd := time.Now()
	fmt.Println(downloadUrl, "File Download Time : ", downEnd.Sub(downStart))

	defer resp.Body.Close()

	// File Read
	ioStart := time.Now()
	var buf bytes.Buffer
	io.Copy(&buf, resp.Body)
	ioEnd := time.Now()
	fmt.Println(downloadUrl, "IO Time Async : ", ioEnd.Sub(ioStart))

	result.Success = true
	imageResult <- result

}

func HttpFileDownLoadWithAsync(downloadUrl string, imageResult chan ImageDownloadResult) {
	result := ImageDownloadResult{
		TargetImagePath: downloadUrl,
	}

	client := http.Client{
		CheckRedirect: func(r *http.Request, via []*http.Request) error {
			r.URL.Opaque = r.URL.Path
			return nil
		},
	}
	// Put content on file
	downStart := time.Now()
	resp, _ := client.Get(downloadUrl)
	downEnd := time.Now()
	fmt.Println(downloadUrl, "File Download Time : ", downEnd.Sub(downStart))

	result.Success = true
	result.Response = resp
	imageResult <- result

}

func getBytesFromDownloadData(resp *http.Response, downloadUrl string) []byte {
	defer func() {
		if err2 := resp.Body.Close(); err2 != nil {
			fmt.Println("Fail Download by image Download Close Error:", downloadUrl)
		}
	}()
	// File Read
	startTime := time.Now()
	var buf bytes.Buffer
	_, err := io.Copy(&buf, resp.Body)

	if err != nil {
		fmt.Println("Fail Download by Read Response Body:", downloadUrl)
		return nil
	}
	fmt.Println(downloadUrl, "IO Time Sync:", time.Now().Sub(startTime))
	return buf.Bytes()
}

以下是日志。

https://myhost.com/item/image/path/name_13.png File Download Time :  197.058394ms
https://myhost.com/item/image/path/name_12.png File Download Time :  399.633804ms
https://myhost.com/item/image/path/name_08.png File Download Time :  587.309339ms
https://myhost.com/item/image/path/name_08.png IO Time Async :  314.482233ms
https://myhost.com/item/image/path/name_03.png File Download Time :  901.985524ms
https://myhost.com/item/image/path/name_05.png File Download Time :  1.132634351s
https://myhost.com/item/image/path/name_02.png File Download Time :  1.132661015s
https://myhost.com/item/image/path/name_14.png File Download Time :  1.132605289s
https://myhost.com/item/image/path/name_09.png File Download Time :  1.132608987s
https://myhost.com/item/image/path/name_16.png File Download Time :  1.133075291s
https://myhost.com/item/image/path/name_01.png File Download Time :  1.132837045s
https://myhost.com/item/image/path/name_11.png File Download Time :  1.133100234s
https://myhost.com/item/image/path/name_10.png File Download Time :  1.132982295s
https://myhost.com/item/image/path/name_07.png File Download Time :  1.133150493s
https://myhost.com/item/image/path/name_12.png IO Time Async :  1.240533838s
https://myhost.com/item/image/path/name_09.png IO Time Async :  849.335303ms
https://myhost.com/item/image/path/name_03.png IO Time Async :  1.080254194s
https://myhost.com/item/image/path/name_02.png IO Time Async :  849.395964ms
https://myhost.com/item/image/path/name_13.png IO Time Async :  1.784857595s
https://myhost.com/item/image/path/name_14.png IO Time Async :  849.642554ms
https://myhost.com/item/image/path/name_16.png IO Time Async :  849.494898ms
https://myhost.com/item/image/path/name_01.png IO Time Async :  850.297187ms
https://myhost.com/item/image/path/name_10.png IO Time Async :  864.482359ms
https://myhost.com/item/image/path/name_11.png IO Time Async :  864.524354ms
https://myhost.com/item/image/path/name_07.png IO Time Async :  874.676604ms
https://myhost.com/item/image/path/name_05.png IO Time Async :  875.22765ms
Total Time Async IO:  2.008162313s
=======================VS========================
https://myhost.com/item/image/path/name_09.png File Download Time :  72.476375ms
https://myhost.com/item/image/path/name_05.png File Download Time :  73.351299ms
https://myhost.com/item/image/path/name_07.png File Download Time :  92.839309ms
https://myhost.com/item/image/path/name_10.png File Download Time :  105.41514ms
https://myhost.com/item/image/path/name_08.png File Download Time :  136.861107ms
https://myhost.com/item/image/path/name_01.png File Download Time :  137.531384ms
https://myhost.com/item/image/path/name_16.png File Download Time :  204.833342ms
https://myhost.com/item/image/path/name_11.png File Download Time :  225.73164ms
https://myhost.com/item/image/path/name_03.png File Download Time :  238.569755ms
https://myhost.com/item/image/path/name_09.png IO Time Sync: 251.986344ms
https://myhost.com/item/image/path/name_14.png File Download Time :  473.071003ms
https://myhost.com/item/image/path/name_02.png File Download Time :  523.402477ms
https://myhost.com/item/image/path/name_13.png File Download Time :  523.389256ms
https://myhost.com/item/image/path/name_12.png File Download Time :  523.412647ms
https://myhost.com/item/image/path/name_05.png IO Time Sync: 549.364233ms
https://myhost.com/item/image/path/name_07.png IO Time Sync: 890.004μs
https://myhost.com/item/image/path/name_10.png IO Time Sync: 545.761μs
https://myhost.com/item/image/path/name_08.png IO Time Sync: 229.321μs
https://myhost.com/item/image/path/name_01.png IO Time Sync: 601.996μs
https://myhost.com/item/image/path/name_16.png IO Time Sync: 12.912227ms
https://myhost.com/item/image/path/name_11.png IO Time Sync: 148.432703ms
https://myhost.com/item/image/path/name_03.png IO Time Sync: 336.862μs
https://myhost.com/item/image/path/name_14.png IO Time Sync: 239.328μs
https://myhost.com/item/image/path/name_02.png IO Time Sync: 483.976μs
https://myhost.com/item/image/path/name_13.png IO Time Sync: 215.655μs
https://myhost.com/item/image/path/name_12.png IO Time Sync: 265.376μs
Total Time Sync IO:  1.039298797s

英文:

I am working on IO after downloading an image from a goroutine.

A question arose during testing.

After downloading an image in a Goroutine, I found a case where the IO operation was very slow.

Rather, it downloads the image in the Goroutine
IO operations outside the groutine realm were faster.

May I know why?

Below is the test source code.

type ImageResult struct {
TargetImagePath string
Success         bool
}
type ImageDownloadResult struct {
TargetImagePath string
Response        *http.Response
Success         bool
}
func main() {
downloadUrls := []string{
&quot;https://myhost.com/item/image/path/name_01.png&quot;,
&quot;https://myhost.com/item/image/path/name_02.png&quot;,
&quot;https://myhost.com/item/image/path/name_03.png&quot;,
&quot;https://myhost.com/item/image/path/name_05.png&quot;,
&quot;https://myhost.com/item/image/path/name_07.png&quot;,
&quot;https://myhost.com/item/image/path/name_08.png&quot;,
&quot;https://myhost.com/item/image/path/name_09.png&quot;,
&quot;https://myhost.com/item/image/path/name_10.png&quot;,
&quot;https://myhost.com/item/image/path/name_11.png&quot;,
&quot;https://myhost.com/item/image/path/name_12.png&quot;,
&quot;https://myhost.com/item/image/path/name_13.png&quot;,
&quot;https://myhost.com/item/image/path/name_14.png&quot;,
&quot;https://myhost.com/item/image/path/name_16.png&quot;,
}
startAsyncIO := time.Now()
resultChannel := make(chan ImageResult)
for _, downloadUrl := range downloadUrls {
go HttpFileDownLoadAndIOWithAsync(downloadUrl, resultChannel)
}
for i := 0; i &lt; len(downloadUrls); i++ {
&lt;-resultChannel
}
fmt.Println(&quot;Total Time Async IO: &quot;, time.Now().Sub(startAsyncIO))
fmt.Println(&quot;=======================VS========================&quot;)
startSyncIO := time.Now()
resultChannel2 := make(chan ImageDownloadResult)
for _, downloadUrl := range downloadUrls {
go HttpFileDownLoadWithAsync(downloadUrl, resultChannel2)
}
for i := 0; i &lt; len(downloadUrls); i++ {
result := &lt;-resultChannel2
getBytesFromDownloadData(result.Response, result.TargetImagePath)
}
fmt.Println(&quot;Total Time Sync IO: &quot;, time.Now().Sub(startSyncIO))
}

func HttpFileDownLoadAndIOWithAsync(downloadUrl string, imageResult chan ImageResult) {
result := ImageResult{
TargetImagePath: downloadUrl,
}
client := http.Client{
CheckRedirect: func(r *http.Request, via []*http.Request) error {
r.URL.Opaque = r.URL.Path
return nil
},
}
// Put content on file
downStart := time.Now()
resp, _ := client.Get(downloadUrl)
downEnd := time.Now()
fmt.Println(downloadUrl, &quot;File Download Time : &quot;, downEnd.Sub(downStart))
defer resp.Body.Close()
// File Read
ioStart := time.Now()
var buf bytes.Buffer
io.Copy(&amp;buf, resp.Body)
ioEnd := time.Now()
fmt.Println(downloadUrl, &quot;IO Time Async : &quot;, ioEnd.Sub(ioStart))
result.Success = true
imageResult &lt;- result
}
func HttpFileDownLoadWithAsync(downloadUrl string, imageResult chan ImageDownloadResult) {
result := ImageDownloadResult{
TargetImagePath: downloadUrl,
}
client := http.Client{
CheckRedirect: func(r *http.Request, via []*http.Request) error {
r.URL.Opaque = r.URL.Path
return nil
},
}
// Put content on file
downStart := time.Now()
resp, _ := client.Get(downloadUrl)
downEnd := time.Now()
fmt.Println(downloadUrl, &quot;File Download Time : &quot;, downEnd.Sub(downStart))
result.Success = true
result.Response = resp
imageResult &lt;- result
}
func getBytesFromDownloadData(resp *http.Response, downloadUrl string) []byte {
defer func() {
if err2 := resp.Body.Close(); err2 != nil {
fmt.Println(&quot;Fail Download by image Download Close Error:&quot;, downloadUrl)
}
}()
// File Read
startTime := time.Now()
var buf bytes.Buffer
_, err := io.Copy(&amp;buf, resp.Body)
if err != nil {
fmt.Println(&quot;Fail Download by Read Response Body:&quot;, downloadUrl)
return nil
}
fmt.Println(downloadUrl, &quot;IO Time Sync:&quot;, time.Now().Sub(startTime))
return buf.Bytes()
}

Below is the log.

https://myhost.com/item/image/path/name_13.png File Download Time :  197.058394ms
https://myhost.com/item/image/path/name_12.png File Download Time :  399.633804ms
https://myhost.com/item/image/path/name_08.png File Download Time :  587.309339ms
https://myhost.com/item/image/path/name_08.png IO Time Async :  314.482233ms
https://myhost.com/item/image/path/name_03.png File Download Time :  901.985524ms
https://myhost.com/item/image/path/name_05.png File Download Time :  1.132634351s
https://myhost.com/item/image/path/name_02.png File Download Time :  1.132661015s
https://myhost.com/item/image/path/name_14.png File Download Time :  1.132605289s
https://myhost.com/item/image/path/name_09.png File Download Time :  1.132608987s
https://myhost.com/item/image/path/name_16.png File Download Time :  1.133075291s
https://myhost.com/item/image/path/name_01.png File Download Time :  1.132837045s
https://myhost.com/item/image/path/name_11.png File Download Time :  1.133100234s
https://myhost.com/item/image/path/name_10.png File Download Time :  1.132982295s
https://myhost.com/item/image/path/name_07.png File Download Time :  1.133150493s
https://myhost.com/item/image/path/name_12.png IO Time Async :  1.240533838s
https://myhost.com/item/image/path/name_09.png IO Time Async :  849.335303ms
https://myhost.com/item/image/path/name_03.png IO Time Async :  1.080254194s
https://myhost.com/item/image/path/name_02.png IO Time Async :  849.395964ms
https://myhost.com/item/image/path/name_13.png IO Time Async :  1.784857595s
https://myhost.com/item/image/path/name_14.png IO Time Async :  849.642554ms
https://myhost.com/item/image/path/name_16.png IO Time Async :  849.494898ms
https://myhost.com/item/image/path/name_01.png IO Time Async :  850.297187ms
https://myhost.com/item/image/path/name_10.png IO Time Async :  864.482359ms
https://myhost.com/item/image/path/name_11.png IO Time Async :  864.524354ms
https://myhost.com/item/image/path/name_07.png IO Time Async :  874.676604ms
https://myhost.com/item/image/path/name_05.png IO Time Async :  875.22765ms
Total Time Async IO:  2.008162313s
=======================VS========================
https://myhost.com/item/image/path/name_09.png File Download Time :  72.476375ms
https://myhost.com/item/image/path/name_05.png File Download Time :  73.351299ms
https://myhost.com/item/image/path/name_07.png File Download Time :  92.839309ms
https://myhost.com/item/image/path/name_10.png File Download Time :  105.41514ms
https://myhost.com/item/image/path/name_08.png File Download Time :  136.861107ms
https://myhost.com/item/image/path/name_01.png File Download Time :  137.531384ms
https://myhost.com/item/image/path/name_16.png File Download Time :  204.833342ms
https://myhost.com/item/image/path/name_11.png File Download Time :  225.73164ms
https://myhost.com/item/image/path/name_03.png File Download Time :  238.569755ms
https://myhost.com/item/image/path/name_09.png IO Time Sync: 251.986344ms
https://myhost.com/item/image/path/name_14.png File Download Time :  473.071003ms
https://myhost.com/item/image/path/name_02.png File Download Time :  523.402477ms
https://myhost.com/item/image/path/name_13.png File Download Time :  523.389256ms
https://myhost.com/item/image/path/name_12.png File Download Time :  523.412647ms
https://myhost.com/item/image/path/name_05.png IO Time Sync: 549.364233ms
https://myhost.com/item/image/path/name_07.png IO Time Sync: 890.004&#181;s
https://myhost.com/item/image/path/name_10.png IO Time Sync: 545.761&#181;s
https://myhost.com/item/image/path/name_08.png IO Time Sync: 229.321&#181;s
https://myhost.com/item/image/path/name_01.png IO Time Sync: 601.996&#181;s
https://myhost.com/item/image/path/name_16.png IO Time Sync: 12.912227ms
https://myhost.com/item/image/path/name_11.png IO Time Sync: 148.432703ms
https://myhost.com/item/image/path/name_03.png IO Time Sync: 336.862&#181;s
https://myhost.com/item/image/path/name_14.png IO Time Sync: 239.328&#181;s
https://myhost.com/item/image/path/name_02.png IO Time Sync: 483.976&#181;s
https://myhost.com/item/image/path/name_13.png IO Time Sync: 215.655&#181;s
https://myhost.com/item/image/path/name_12.png IO Time Sync: 265.376&#181;s
Total Time Sync IO:  1.039298797s

答案1

得分: 3

Go代码总是在goroutine中执行。

Goroutine是相等的，它们没有区别（除非运行main()函数的main goroutine结束，整个应用程序才会终止）。

Goroutine被调度/复用到操作系统线程上，线程在物理或虚拟CPU核心上运行。因此，一个goroutine运行得比另一个慢/快取决于执行其指令的CPU核心的负载情况（系统级）。执行一个goroutine的CPU核心可能比另一个核心更多地被利用，导致较差的感知goroutine性能，但这不是因为Go的运行时和goroutine调度。

请注意，可以使用runtime.LockOSThread()将goroutine锁定到操作系统线程，这意味着该goroutine将“拥有”该线程（没有其他goroutine将被调度到该线程上），但您的应用程序中没有使用它。

因此，您在其他goroutine中经历较慢的下载与Go无关，可能与您的操作系统和CPU负载或您调用的外部服务（HTTP服务器）有关。

还要注意，再次运行相同的代码可能需要更少的时间，初始化的代码可以重用，并且连接到相同的主机也可能快得多：DNS查找被缓存，甚至TCP连接也可以被缓存/池化。您调用的服务器也可能缓存某些数据，因此获取相同的URL可能也会快得多（从同一服务器获取不同资源在后续调用中也可能更快，某些检查，如身份验证/授权可能会被缓存）。

参考链接：https://stackoverflow.com/questions/41608578/order-of-the-code-and-performance/41608707#41608707

英文:

Go code is always executed in goroutines.

Goroutines are equal, they are not distinguished (except when the main goroutine running the main() function ends, the whole app is terminated).

Goroutines are scheduled / multiplexed onto OS threads, and threads run on physical or virtual CPU cores. So whether one goroutine runs slower / faster than another depends on how the CPU core executing its instructions is loaded (system-wise). A CPU core eventually executing one goroutine's instructions may be utilized more than another, resulting in worse perceptived goroutine performance, but this is not because of Go's runtime and goroutine scheduling.

Note that it's possible to lock a goroutine to an OS thread using runtime.LockOSThread() which means the goroutine will "own" that thread (no other goroutine will be scheduled onto that thread), but you don't use it in your app.

So you experiencing slower download from other goroutines is not Go related, it may relate to your OS and CPU load or the external service (HTTP server) you call.

Also note that running the same code again may take significantly less time, initialized code may be reused, and connecting to the same host may also be significantly faster: DNS lookups are cached, even TCP connections may be cached / pooled. The server you call may also cache certain data, so fetching the same URLs may also be significantly faster (fetching different resources from the same server may also be faster in subsequent calls, certain checks like authentication / authorization may be cached).

通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。

为什么 Goroutines 中的 IO 操作速度较慢？

问题

答案1

使用Go语言的WithTimeout函数如何检测超时发生

这段代码中的TCP连接何时关闭？

How to run mockgen during build?

使用sqlx进行结构嵌入，但无法从数据库返回值。

What's the correct way to type hint an empty list as a literal in python?

如何在Highcharts Gantt中更改本地化的星期名称

如何在同一个流中使用多个过滤器和映射函数？

如何使用Map/Set来将代码优化到O(n)？

.NET MAUI Android在GitHub Actions上构建失败，错误代码为1。

如何在Playwright视觉比较中屏蔽多个定位器？

在C++中，可以使用可变模板参数来检索类型的内部类型。

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found

Creating and opening a URL to log in to Website via Basic Auth with Robot Framework/Selenium (Python)

AG Grid 在上下文菜单中以大文本形式打开

发表评论