Goroutines vs asyncio tasks + 线程池用于 CPU 密集型调用

huangapple go评论82阅读模式
英文:

Goroutines vs asyncio tasks + thread pool for CPU-bound calls

问题

Goroutines大致相当于Python的asyncio任务,但有一个额外的特性,即任何CPU密集型任务都会被路由到ThreadPoolExecutor而不是添加到事件循环中(当然,前提是我们使用没有GIL的Python解释器)。

除了并发作为Go的一个固有部分带来的效率和代码清晰度之外,我是否忽略了两种方法之间的任何实质性差异?

英文:

Are goroutines roughly equivalent to python's asyncio tasks, with an additional feature that any CPU-bound task is routed to a ThreadPoolExecutor instead of being added to the event loop (of course, with the assumption that we use a python interpreter without GIL)?

Is there any substantial difference between the two approaches that I'm missing? Of course, apart from the efficiencies and code clarity that result from the concurrency being an integral part of Go.

答案1

得分: 12

我认为我知道部分答案。我试图总结我对asyncio任务和goroutine之间的区别的理解,按重要性排序:

1)与asyncio不同,人们很少需要担心他们的goroutine会阻塞太久。然而,与asyncio任务相比,goroutine之间的内存共享更类似于线程之间的内存共享,因为goroutine的执行顺序保证要弱得多(即使硬件只有一个核心)。

asyncio只会在显式的awaityield和某些事件循环方法上切换上下文,而Go运行时可能会在更微妙的触发器上切换上下文(例如某些函数调用)。因此,asyncio是完全协作的,而goroutine只是大部分协作(而且路线图表明它们将来会变得更不协作)。

一个非常紧密的循环(例如数值计算)仍然可能会阻塞Go运行时(以及它所在的线程)。如果发生这种情况,它对性能的影响将比在Python中小得多,除非它发生在多个线程中。

2)Goroutine具有开箱即用的并行计算支持,而在asyncio下则需要更复杂的方法。

Go运行时可以并行运行线程(如果有多个核心可用),因此它有点类似于在没有GIL的Python运行时下在线程池中运行多个asyncio事件循环,并在前面使用一个语言感知的负载均衡器。

3)Go运行时会自动在单独的线程中处理阻塞的系统调用;而在asyncio下,这需要显式地完成(例如使用run_in_executor)。

话虽如此,在内存成本方面,goroutine与asyncio任务非常相似,而不是线程。

英文:

I think I know part of the answer. I tried to summarize my understanding of the differences, in order of importance, between asyncio tasks and goroutines:

  1. Unlike under asyncio, one rarely needs to worry that their goroutine will block for too long. OTOH, memory sharing across goroutines is akin to memory sharing across threads rather than asyncio tasks since goroutine execution order guarantees are much weaker (even if the hardware has only a single core).

asyncio will only switch context on explicit await, yield and certain event loop methods, while Go runtime may switch on far more subtle triggers (such as certain function calls). So asyncio is perfectly cooperative, while goroutines are only mostly cooperative (and the roadmap suggests they will become even less cooperative over time).

A really tight loop (such as with numeric computation) could still block Go runtime (well, the thread it's running on). If it happens, it's going to have less of an impact than in python - unless it occurs in mutliple threads.

  1. Goroutines are have off-the-shelf support for parallel computation, which would require a more sophisticated approach under asyncio.

Go runtime can run threads in parallel (if multiple cores are available), and so it's somewhat similar to running multiple asyncio event loops in a thread pool under a GIL-less python runtime, with a language-aware load balancer in front.

  1. Go runtime will automatically handle blocking syscalls in a separate thread; this needs to be done explicitly under asyncio (e.g., using run_in_executor).

That said, in terms of memory cost, goroutines are very much like asyncio tasks rather than threads.

答案2

得分: 1

我想你可以这样认为它在底层是这样工作的,当然。虽然不是非常准确,但足够接近。

但是有一个很大的区别:在Go中,你可以编写直线代码,所有的I/O阻塞都会自动处理。你可以在简单的直线代码中调用Read,然后Write,再调用Read。而在Python的asyncio中,据我了解,你需要排队一个函数来处理读取,而不仅仅是调用Read。

英文:

I suppose you could think of it working that way underneath, sure. It's not really accurate, but, close enough.

But there is a big difference: in Go you can write straight line code, and all the I/O blocking is handled for you automatically. You can call Read, then Write, then Read, in simple straight line code. With Python asyncio, as I understand it, you need to queue up a function to handle the reads, rather than just calling Read.

huangapple
  • 本文由 发表于 2017年5月31日 06:34:33
  • 转载请务必保留本文链接:https://go.coder-hub.com/44272954.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定