英文:
Sync Map possibly leading increase in ram and goroutines
问题
你好,这是我翻译好的代码部分:
import (
"context"
"errors"
"sync"
"time"
)
type Collector struct {
keyValMap *sync.Map
}
func (c *Collector) LoadOrWait(key any) (retValue any, availability int, err error) {
value, status := c.getStatusAndValue(key)
switch status {
case 0:
return nil, 0, nil
case 1:
return value, 1, nil
case 2:
ctxWithTimeout, _ := context.WithTimeout(context.Background(), 5 * time.Second)
for {
select {
case <-ctxWithTimeout.Done():
return nil, 0, errRequestTimeout
default:
value, resourceStatus := c.getStatusAndValue(key)
if resourceStatus == 1 {
return value, 1, nil
}
time.Sleep(50 * time.Millisecond)
}
}
}
return nil, 0, errRequestTimeout
}
// Store ...
func (c *Collector) Store(key any, value any) {
c.keyValMap.Store(key, value)
}
func (c *Collector) getStatusAndValue(key any) (retValue any, availability int) {
var empty any
result, loaded := c.keyValMap.LoadOrStore(key, empty)
if loaded && result != empty {
return result, 1
}
if loaded && result == empty {
return empty, 2
}
return nil, 0
}
这个工具的目的是作为一个缓存,只加载一次相似的值,但可以多次读取。然而,当将Collector对象传递给多个goroutine时,当多个goroutine尝试使用collector缓存时,我发现goroutine的数量和内存使用量增加。有人能解释一下sync.Map的使用是否正确吗?如果是的话,可能导致goroutine数量增加/内存使用量增加的原因是什么?
英文:
Hi here is the code where I make util called as collector
import (
"context"
"errors"
"sync"
"time"
)
type Collector struct {
keyValMap *sync.Map
}
func (c *Collector) LoadOrWait(key any) (retValue any, availability int, err error) {
value, status := c.getStatusAndValue(key)
switch status {
case 0:
return nil, 0, nil
case 1:
return value, 1, nil
case 2:
ctxWithTimeout, _ := context.WithTimeout(context.Background(), 5 * time.Second)
for {
select {
case <-ctxWithTimeout.Done():
return nil, 0, errRequestTimeout
default:
value, resourceStatus := c.getStatusAndValue(key)
if resourceStatus == 1 {
return value, 1, nil
}
time.Sleep(50 * time.Millisecond)
}
}
}
return nil, 0, errRequestTimeout
}
// Store ...
func (c *Collector) Store(key any, value any) {
c.keyValMap.Store(key, value)
}
func (c *Collector) getStatusAndValue(key any) (retValue any, availability int) {
var empty any
result, loaded := c.keyValMap.LoadOrStore(key, empty)
if loaded && result != empty {
return result, 1
}
if loaded && result == empty {
return empty, 2
}
return nil, 0
}
So the purpose of this util is to act as a cache where similar value is only loaded once but read many times. However when an object of Collector is passed to multiple goroutines I am facing increase in gorotines and ram usage whenever multiple goroutines are trying to use collector cache. Could someone explain if this usage of sync Map is correct. If yes then what might be the cause high number of goroutines / high ram usage
答案1
得分: 1
肯定的是,由于没有调用新创建的ctxWithTimeout
上下文的cancel函数,你可能面临内存泄漏的问题。为了解决这个问题,将代码改为以下形式:
ctxWithTimeout, cancelFunc := context.WithTimeout(context.Background(), requestTimeout)
defer cancelFunc()
通过这样做,一旦上下文过期,你就可以确保清理所有分配的资源。这应该解决泄漏问题。
关于使用sync.Map
,我认为是可以的。
如果这解决了你的问题,或者还有其他问题需要解决,请告诉我,谢谢!
英文:
For sure, you're facing possible memory leaks due to not calling the cancel func of the newly created ctxWithTimeout
context. In order to fix this change the line to these:
ctxWithTimeout, cancelFunc := context.WithTimeout(context.Background(), requestTimeout)
defer cancelFunc()
Thanks to this, you're always sure to clean up all the resources allocated once the context expires. This should address the issue of the leaks.
About the usage of sync.Map
seems good to me.
Let me know if this solves your issue or if there is something else to address, thanks!
答案2
得分: 1
你在读者端展示了代码,但没有展示发送请求的代码(调用.Store(key, value)
的代码)。
根据你展示的代码:
- 第一个尝试访问给定键的 goroutine 会将你的
empty
值存储在映射中(执行c.keyValMap.LoadOrStore(key, empty)
), - 因此,后续查询相同键的所有 goroutine 都会进入“带超时的查询”循环,即使实际执行请求并将其结果存储在缓存中的操作并未执行。
[在你更新之后]
就你的收集器代码本身而言,资源消耗似乎还可以:我在这段代码中没有看到死锁或者 goroutine 的增加。
你可能应该查看代码中的其他地方。
此外,如果这个结构只增长而不缩小,它将消耗更多的内存。请审查你的程序,评估缓存中可以存放多少个不同的键以及缓存值可以占用多少内存。
英文:
You show the code on the reader side of things, but not the code which does the request (and calls .Store(key, value)
).
With the code you display :
- the first goroutine which tries to access a given key will store your
empty
value in the map (when executingc.keyValMap.LoadOrStore(key, empty)
), - so all goroutines that will come afterwards querying for the same key will enter the "query with timeout" loop -- even if the action that actually runs the request and stores its result in the cache isn't executed.
[after your update]
The code for your collector alone seems to be ok regarding resource consumption : I don't see deadlocks or multiplication of goroutines in that code alone.
You should probably look at other places in your code.
Also, if this structure only grows and never shrinks, it is bound to consume more memory. Do audit your program to evaluate how many different keys can live together in your cache and how much memory the cached values can occupy.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论