CoInitializeEx(COINIT_MULTITHREADED)和使用WMI的Goroutines

huangapple go评论72阅读模式
英文:

CoInitializeEx(COINIT_MULTITHREADED) and Goroutines using WMI

问题

我们有一个用Go编写的监控代理程序,它使用多个goroutine从WMI收集系统指标。最近我们发现,当在Server 2016或Windows 10上运行go二进制文件时(可能也适用于使用WMF 5.1的其他操作系统),程序会出现内存泄漏。在创建了一个最小化的测试用例来重现这个问题之后,似乎只有在对ole.CoInitializeEx方法进行大量调用时才会发生内存泄漏(可能是在WMF 5.1中发生了某些变化,但我们无法在同一系统上使用python comtypes包重现这个问题)。

我们在应用程序中使用COINIT_MULTITHREADED来进行多线程公寓(MTA),我的问题是:因为我们从各个goroutine发出OLE/WbemScripting调用,我们需要在启动时只调用一次ole.CoInitializeEx还是在每个goroutine中调用一次?我们的查询代码已经使用runtime.LockOSThread来防止调度程序在不同的操作系统线程上运行该方法,但是CoInitializeEx的MSDN备注似乎表明它必须在每个线程上至少调用一次。我不知道有没有办法确保新的goroutine在已初始化的操作系统线程上运行,因此多次调用CoInitializeEx似乎是正确的方法(并且在过去几年中一直工作正常)。

我们已经重构了代码,将所有的WMI调用都放在一个专用的后台工作线程中,但我很想知道我们的原始代码是否应该只在启动时使用一个CoInitializeEx,而不是每个goroutine都调用一次。

英文:

We have a monitoring agent written in Go that uses a number of goroutines to gather system metrics from WMI. We recently discovered the program was leaking memory when the go binary is run on Server 2016 or Windows 10 (also possibly on other OS using WMF 5.1). After creating a minimal test case to reproduce the issue it seems that the leak only occurs if you make a large number of calls to the ole.CoInitializeEx method (possibly something changed in WMF 5.1 but we could not reproduce the issue using the python comtypes package on the same system).

We are using COINIT_MULTITHREADED for multithread apartment (MTA) in our application, and my question is this: Because we are issuing OLE/WbemScripting calls from various goroutines, do we need to call ole.CoInitializeEx just once on startup or once in each goroutine? Our query code already uses runtime.LockOSThread to prevent the scheduler from running the method on different OS threads, but the MSDN remarks on CoInitializeEx seem to indicate it must be called at least once on each thread. I am not aware of any way to make sure new goroutines run on an already initialized OS thread, so multiple calls to CoInitializeEx seemed like the correct approach (and worked fine for the last few years).

We have already refactored the code to do all the WMI calls on a dedicated background worker, but I am curious to know if our original code should work using only one CoInitializeEx at startup instead of once for every goroutine.

答案1

得分: 1

据我所知,由于Win32 API仅根据本机操作系统线程定义,对CoInitialize[Ex]()的调用只会影响完成调用的线程。

由于Go运行时使用自由的M×N调度将goroutine映射到操作系统线程,并且这些线程在运行时根据需要动态创建/删除,对于确保CoInitialize[Ex]()调用对其所在的goroutine产生持久影响的唯一方法是通过调用runtime.LockOSThread()将该goroutine绑定到其当前的操作系统线程,并对每个打算进行COM调用的goroutine执行此操作。

请注意,这基本上创建了goroutine和操作系统线程之间的1×1映射,这在很大程度上破坏了goroutine的初衷。因此,你可能希望考虑只有一个goroutine调用COM并在通道上监听请求,或者在另一个goroutine的后面隐藏一个这样的工作goroutine池,该goroutine将客户端的请求分派给工作goroutine。

关于COINIT_MULTITHREADED的更新

引用文档

> 多线程(也称为自由线程)允许在此线程上创建的对象的方法调用在任何线程上运行。没有调用的序列化 - 可以对同一方法或同一对象进行多次调用,或同时进行多次调用。多线程对象并发提供了最高的性能,并且对于跨线程、跨进程和跨机器调用,最好利用多处理器硬件,因为对象的调用不以任何方式进行序列化。然而,这意味着对象的代码必须通过使用同步原语(如临界区、信号量或互斥量)来强制执行自己的并发模型。此外,由于对象无法控制访问它的线程的生命周期,因此不能在对象中存储线程特定状态(在线程本地存储中)。

因此,COM线程模型与线程本身的初始化无关,而是与COM子系统如何允许调用在COM初始化的线程上创建的COM对象的方法有关。

如果你将一个线程初始化为COINIT_MULTITHREADED并在其上创建一些COM对象,然后将其引用传递给该对象的某个外部客户端,以便该客户端能够调用该对象的方法,操作系统可以在进程中的任何线程上调用这些方法。

我真的不知道这应该如何与Go运行时交互,所以我会从小处开始,使用单个STA模型的线程,然后如果需要的话再尝试使其更复杂。

另一方面,如果你只实例化外部COM对象而不将其描述符传递给外部(似乎是这种情况),则线程模型应该不相关。也就是说,除非WUA API中的某些代码调用了你实例化的COM对象上的某个“事件”方法。

英文:

AFAIK, since Win32 API is defined only in terms of native OS threads, a call to CoInitialize[Ex]() only ever affects the thread it completed on.

Since the Go runtime uses free M×N scheduling of the goroutines to OS threads, and these threads are created / deleted as needed at runtime in a manner completely transparent to the goroutines, the only way to make sure the CoInitialize[Ex]() call has any lasting effect on the goroutine it was performed on is to first bind that goroutine to its current OS thread by calling runtime.LockOSThread() and doing this for every goroutine intended to do COM calls.

Please note that this basically creates an 1×1 mapping between goroutines and OS threads which defeats much of the purpose of goroutines to begin with. So supposedly you might want to consider having just a single goroutine calling into COM and listening for requests on a channel, or having
a pool of such worker goroutines hidden behing another one which would dispatch the clients' requests onto the workers.

Update regarding COINIT_MULTITHREADED.

To cite the docs:

> Multi-threading (also called free-threading) allows calls to methods
> of objects created by this thread to be run on any thread. There is no
> serialization of calls — many calls may occur to the same method or
> to the same object or simultaneously. Multi-threaded object
> concurrency offers the highest performance and takes the best
> advantage of multiprocessor hardware for cross-thread, cross-process,
> and cross-machine calling, since calls to objects are not serialized
> in any way. This means, however, that the code for objects must
> enforce its own concurrency model, typically through the use of
> synchronization primitives, such as critical sections, semaphores, or
> mutexes. In addition, because the object doesn't control the lifetime
> of the threads that are accessing it, no thread-specific state may be
> stored in the object (in Thread Local Storage).

So basically the COM threading model has nothing to do with initialization of the threads theirselves—but rather with how the COM subsystem is allowed to call the methods of the COM objects you create on the COM-initialized threads.

IIUC, if you will COM-initialize a thread as COINIT_MULTITHREADED and create some COM object on it, and then pass its reference to some outside client of that object so that it is able to call that object's methods, those methods can be called by the OS on any thread in your process.

I really have no idea how this is supposed to interact with Go runtime,
so I'd start small and would use a single thread with STA model and then
maybe try to make it more complicated if needed.

On the other hand, if you only instantiate external COM objects and not
transfer their descriptors outside (and it appears that's the case),
the threading model should not be relevant. That is, only unless some
code in the WUA API would call some "event-like" method on a COM object you
have instantiated.

huangapple
  • 本文由 发表于 2017年3月2日 10:26:06
  • 转载请务必保留本文链接:https://go.coder-hub.com/42545728.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定