Golang:内存问题困扰。

huangapple go评论74阅读模式
英文:

golang : trouble with memory

问题

我有内存问题。当我的程序运行很长时间时,我不明白为什么Go语言会越来越多地使用内存(从不释放)。

在第一次分配后,程序使用了近9 MB的内存。然后经过12个小时后,它开始以指数方式使用更多的内存,直到达到800 MB。

对于每个调用,我的服务需要使用接收到的参数生成一个新的模板,正如你所看到的,我为每个调用创建了一个新的端点。我不知道这是否是一个好主意,我认为问题可能来自代码的这部分,但我不确定,因为我不知道Go语言是如何管理它的。

英文:

I have trouble with memory. I don't understand why Go uses more and more memory (never freeing it) when my program runs for a long time.

After the first allocation, program uses nearly 9 MB of memory. Then after 12 hours it starts to use more memory exponentially, until 800 MB.

//.....code.....
if bol {
    // Assignment Struct.Var
    Struct_VastScript.TxtNoticeTop = JsonStruct_S.Options.TxtNoticeTop
    Struct_VastScript.TxtNoticeBottom = JsonStruct_S.Options.TxtNoticeBottom
    Struct_VastScript.Loop = JsonStruct_S.Options.Loop

    Struct_Image, err := getImage(Struct_VastScript.Video)
    if err == nil {
        if mobile == "true" {
            Struct_VastScript.Image = Struct_Image.URL360
        }
    }
    //open and parse a template file
    fi = path.Join("templates/VastPlayer", "TempVastPlayer.txt")
    tmpl, err := template.ParseFiles(fi)

    if err != nil {
        job_1.Complete(health.Panic)
        return false, err
    }
    //substitute fields in the template 'tmpl', with values from 'XmlStruct_V' and write it out to 'buf'
    var buf bytes.Buffer
    if err := tmpl.Execute(&buf, Struct_VastScript); err != nil {
        //if  err := tmpl.Execute(w, XmlStruct_V); err != nil {
        job_1.Complete(health.Panic)
        return false, err
    }

    // Call Func randString() : return alphanum random
    dir := randString(12)
    fpath := "http://creative2.xxx.io/api/html/" + dir

    // Create a new EndPoint to write the generated 'template' on 'w' http.ResponseWriter
    routeHtml := "/api/html/" + dir
    http.HandleFunc(routeHtml, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        //writes Template to 'w' http.ResponseWriter
        fmt.Fprintf(w, buf.String())
        fmt.Println("successfull Operation 2 !!")
        fmt.Println("")
        job_2.Complete(health.Success)
    }))


    //Call Func JsonReply(): return the finale Json response
    str := JsonReply(fpath, JsonStruct_S.Options.Animated, JsonStruct_S.Options.Responsive, JsonStruct_S.Options.Clickurl, JsonStruct_S.Options.Width, JsonStruct_S.Options.Height, adid, campaignid, JsonStruct_S.Type, JsonStruct_S.Options.Aspectratio, mobile)
    w.Header().Set("Content-Type", "application/json")
    //writes FinaleJson to 'w' http.ResponseWriter(it contains the link of the second endpoint "/api/html/")
    fmt.Fprint(w, str)
    fmt.Println("successfull Operation !!")
    fmt.Println("")
    job_1.Complete(health.Success)
    return true, nil
} else {
    return false, nil
}

For each call,my service need to generate a new template with the params that I receive,as you see I create a new endpoint for each call, I don't know if it's a good idea, I think the problem comes from this part of code but Im not sure because I don't know how GO manage it.

答案1

得分: 3

显然,每次请求出现时都不应该创建处理程序。它们从不释放内存,所以最终会导致内存溢出异常。

相反,将处理程序的端点放入数组(切片)中,并使用一个处理程序来响应请求,该处理程序通过在此切片中查找URL并将其从切片中删除,以便不再需要。

所以基本上,不要这样做:

routeHtml := "/api/html/" + dir
http.HandleFunc(routeHtml, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    //将模板写入'w' http.ResponseWriter
    fmt.Fprintf(w, buf.String())
    fmt.Println("成功操作2!!")
    fmt.Println("")
    job_2.Complete(health.Success)
}))

而是这样做:

type JobInfo struct {
    Path string
    // 这里可能有一些数据
}

// 可能是全局上下文
var jobs []JobInfo

// 初始化
jobs = make([]JobInfo, 0)

http.HandleFunc("/api/html/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    path := r.URL.Path
    var job *JobInfo
    for _, j := range jobs {
        if j.Path == path {
            job = &j
            break
        }
    }

    if job != nil {
        // 在这里处理作业请求
    }
}))

// 然后在作业循环中
handlers = append(handlers, JobInfo{"/api/html/" + dir, ...})

它会起作用(因为):

> 模式名称固定,根路径,如“/favicon.ico”,或根子树,如“/images/”(注意尾部斜杠)。较长的模式优先于较短的模式,因此,如果对“/images/”和“/images/thumbnails/”都注册了处理程序,后者将用于以“/images/thumbnails/”开头的路径,而前者将接收到“/images/”子树中的任何其他路径的请求。

当然不要忘记清理数组jobs

英文:

Obviously, you should not create handler every time request appears. They never free the memory so you will end up having out of memory exception.

Instead, put the handler endpoint into array (slice) and use ONE handler that responds to the request by looking the URL in this slice and then removing the item from the slice with it is not needed any longer.

So basically, instead of

routeHtml := "/api/html/" + dir
http.HandleFunc(routeHtml, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    //writes Template to 'w' http.ResponseWriter
    fmt.Fprintf(w, buf.String())
    fmt.Println("successfull Operation 2 !!")
    fmt.Println("")
    job_2.Complete(health.Success)
}))

do

type JobInfo struct {
    Path string
    // some data here
}

// maybe global context
var jobs []JobInfo

// initialisation
jobs = make([]JobInfo, 0)

http.HandleFunc("/api/html/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    path := r.URL.Path
    var job *JobInfo
    for _, j := range jobs {
        if j.Path == path {
            job = &j
            break
        }
    }

    if job != nil {
        // handle job request here
    }
}))

// and then in the jobs' loop
handlers = append(handlers, JobInfo{"/api/html/" + dir, ...})

It will work because:

> Patterns name fixed, rooted paths, like "/favicon.ico", or rooted subtrees, like "/images/" (note the trailing slash). Longer patterns take precedence over shorter ones, so that if there are handlers registered for both "/images/" and "/images/thumbnails/", the latter handler will be called for paths beginning "/images/thumbnails/" and the former will receive requests for any other paths in the "/images/" subtree.

Do not forget to clean the array jobs, of course.

答案2

得分: 0

使用map而不是slice更好。

type JobInfo struct {
    Path string
    // 这里是一些数据
}

// 全局上下文
var jobs map[string]JobInfo

// 初始化
jobs = make(map[string]JobInfo)

http.HandleFunc("/api/html/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    path := r.URL.Path
    var job JobInfo
    var ok bool

    job, ok = jobs[path]
    if ok {
        // 在这里处理作业请求
        // 然后删除作业
        delete(jobs, path)
    }

}))

// 然后在作业循环中
pathVideo := "/api/html/" + dir
jobs[pathVideo] = JobInfo{Path: pathVideo, ...}

希望对你有帮助!

英文:

Instead of using slice it's better to use map

type JobInfo struct {
    Path string
    // some data here
}
    
    // global context
    var jobs map[string]JobInfo
    
// initialisation
jobs = make(map[string]JobInfoStruct)
    
 
http.HandleFunc("/api/html/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    path := r.URL.Path
    var job JobInfoStruct
    var ok bool
    
    job, ok = jobs[path]
    if ok {
      // handle job request here
      //then after delete the job
      delete(jobs, path)
    }
    
}))
    
 // and then in the jobs' loop
pathVideo := "/api/html/" + dir
jobs[pathVideo] = JobInfoStruct{pathVideo, ...}

huangapple
  • 本文由 发表于 2015年7月30日 18:04:39
  • 转载请务必保留本文链接:https://go.coder-hub.com/31720810.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定