使用多个接收器来接收一个频道的内容

huangapple go评论78阅读模式
英文:

Using multiple receivers for a channel

问题

我正在尝试使用go-json-rest在golang中编写一个REST服务。

该服务的目的只是将接收到的数据转换为CSV格式并记录下来。
由于负载可能很重,我希望使用goroutines进行日志记录。
目前我已经创建了四个LogWorker(goroutine)
每个goroutine将CSV记录到单独的文件中。

当我执行代码时,日志总是从最后一个goroutine触发。我在我的日志文件夹中只看到一个文件,来自第四个routine。

这是我的服务器代码:

package main

import (
    "github.com/ant0ine/go-json-rest/rest"
    "log"
    "net/http"
    "strconv"
    "time"
)

const workerCount = 4
var evChannel = make(chan Event)
var workers = make([]*LogWorker, workerCount)
const maxLogFileSize = 100 // In MB
const maxLogFileBackups = 30
const maxLogFileAge = 5
const logFileName = "/home/sam/tmp/go_logs/event_"

func main() {
    // 初始化 workers
    // 创建了四个 workers
    for i := 0; i < workerCount; i++ {
        var fileName = logFileName + strconv.Itoa(i)
        workers[i] = NewLogWorker(fileName, maxLogFileSize, maxLogFileBackups, maxLogFileAge)
        go workers[i].Work(evChannel)
    }    

    // 初始化 REST API
    api := rest.NewApi()
    //api.Use(rest.DefaultDevStack...)
    api.Use(rest.DefaultCommonStack...)
    router, err := rest.MakeRouter(
        rest.Post("/events", StoreEvents),
    )
    if err != nil {
        log.Fatal(err)
    }
    api.SetApp(router)
    log.Fatal(http.ListenAndServe(":4545", api.MakeHandler()))
}

func StoreEvents(w rest.ResponseWriter, r *rest.Request) {
    event := Event{}
    err := r.DecodeJsonPayload(&event)
    if err != nil {
        rest.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    // TODO : Add validation if needed
    // Add code to parse the request and add further information to event 
    // log.Println()
    select {
        case evChannel <- event:
        case <- time.After(5 * time.Second):
      // throw away the message, so sad
    }    
    // evChannel <- event
    //log.Println(Csv(event))
    w.WriteHeader(http.StatusOK)
}

这是我的worker代码:

package main

import (
    "gopkg.in/natefinch/lumberjack.v2"
    "log"
    "fmt"
)

type LogWorker struct {
    FileName string
    MaxSize int // In megabytes
    MaxBackups int // No of backups per worker
    MaxAge int // maximum number of days to retain old log files
}

func NewLogWorker(fileName string, maxSize int, maxBackups int, maxAge int) (lw *LogWorker) {
    return &LogWorker {fileName, maxSize, maxBackups, maxAge}
}

func (lw *LogWorker) Work(evChannel chan Event) {
    fmt.Println(lw.FileName)
    log.SetOutput(&lumberjack.Logger {
        Filename:   lw.FileName,
        MaxSize:    lw.MaxSize,
        MaxBackups: lw.MaxBackups,
        MaxAge:     lw.MaxAge,
    })
    log.SetFlags(0)

    for {
        event := <- evChannel
        log.Println(Csv(event))
    }
}

请注意,event是一个包含一些字符串字段的结构体。
SO上已经有一个类似的问题。当我尝试在playground中执行goroutine时,它仍然打印出最后一个goroutine的值。提供的答案中有一些wait.Done。由于我的worker需要持续运行,我认为我不能使用它。

请帮助我找出为什么我的所有goroutines(LogWorkers)都没有被使用?

英文:

I am trying write a REST service in golang using go-json-rest

The purpose of the service is just to convert the received data to CSV and log it.
Since the load may be heavy, I would like to do the logging using goroutines.
Currently I have created four LogWorkers(goroutine)
Each goroutine will log the CSV into separate files.

When I execute the code, the log is always triggered from the last goroutine. I see a single file created in my log folder which is from fourth routine.

Here is my server code

package main
import (
&quot;github.com/ant0ine/go-json-rest/rest&quot;
&quot;log&quot;
&quot;net/http&quot;
&quot;strconv&quot;
&quot;time&quot;
)
const workerCount = 4
var evChannel = make(chan Event)
var workers = make([]*LogWorker, workerCount)
const maxLogFileSize = 100 // In MB
const maxLogFileBackups = 30
const maxLogFileAge = 5
const logFileName = &quot;/home/sam/tmp/go_logs/event_&quot;
func main() {
// Initialize workers
// Four workers is being created
for i := 0; i &lt; workerCount; i++ {
var fileName = logFileName + strconv.Itoa(i)
workers[i] = NewLogWorker(fileName, maxLogFileSize, maxLogFileBackups, maxLogFileAge)
go workers[i].Work(evChannel)
}    
// Initialize REST API
api := rest.NewApi()
//api.Use(rest.DefaultDevStack...)
api.Use(rest.DefaultCommonStack...)
router, err := rest.MakeRouter(
rest.Post(&quot;/events&quot;, StoreEvents),
)
if err != nil {
log.Fatal(err)
}
api.SetApp(router)
log.Fatal(http.ListenAndServe(&quot;:4545&quot;, api.MakeHandler()))
}
func StoreEvents(w rest.ResponseWriter, r *rest.Request) {
event := Event{}
err := r.DecodeJsonPayload(&amp;event)
if err != nil {
rest.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// TODO : Add validation if needed
// Add code to parse the request and add further information to event 
// log.Println()
select {
case evChannel &lt;- event:
case &lt;- time.After(5 * time.Second):
// throw away the message, so sad
}    
// evChannel &lt;- event
//log.Println(Csv(event))
w.WriteHeader(http.StatusOK)
}

here is my worker code

package main
import (
&quot;gopkg.in/natefinch/lumberjack.v2&quot;
&quot;log&quot;
&quot;fmt&quot;
)
type LogWorker struct {
FileName string
MaxSize int // In megabytes
MaxBackups int // No of backups per worker
MaxAge int // maximum number of days to retain old log files
}
func NewLogWorker(fileName string, maxSize int, maxBackups int, maxAge int) (lw *LogWorker) {
return &amp;LogWorker {fileName, maxSize, maxBackups, maxAge}
}
func (lw *LogWorker) Work(evChannel chan Event) {
fmt.Println(lw.FileName)
log.SetOutput(&amp;lumberjack.Logger {
Filename:   lw.FileName,
MaxSize:    lw.MaxSize,
MaxBackups: lw.MaxBackups,
MaxAge:     lw.MaxAge,
})
log.SetFlags(0)
for {
event := &lt;- evChannel
log.Println(Csv(event))
}
}

Please note that event is a struct which contains some string fields.
Already there is a similar question in SO. When I tried to execute the goroutine in playground, it still prints the value from last go routine. The answer provided has some wait.Done. As my worker needs to run continuously I don't I think I can use it.

Please help me to find why my all goroutines (LogWorkers) are not used?

答案1

得分: 4

你正在为每个goroutine设置日志包的默认全局记录器的输出。

你可能想要做更多的事情,像这样:

func (lw *LogWorker) Work(evChannel chan Event) {
    fmt.Println(lw.FileName)
    lg := log.New(&lumberjack.Logger {
        Filename:   lw.FileName,
        MaxSize:    lw.MaxSize,
        MaxBackups: lw.MaxBackups,
        MaxAge:     lw.MaxAge,
    }, "", 0)

    for {
        event := <- evChannel
        lg.Println(Csv(event))
    }
}

这样将为每个goroutine提供一个记录器。

在你的版本中,你可能只有最后一个执行的记录器(可能是最后一个生成的goroutine,但不能保证)。

为了进一步改进,你可能还想将你的for循环写成:

for event := range evChannel {
    lg.Println(Csv(event))
}

这样,当通道关闭时,它将终止goroutine,而不是在关闭的通道中旋转空值。参考这里

英文:

You are setting the log package's default global logger's output in each goroutine.

You probably want to do something more like:

func (lw *LogWorker) Work(evChannel chan Event) {
fmt.Println(lw.FileName)
lg := log.New(&amp;lumberjack.Logger {
Filename:   lw.FileName,
MaxSize:    lw.MaxSize,
MaxBackups: lw.MaxBackups,
MaxAge:     lw.MaxAge,
}, &quot;&quot;, 0)
for {
event := &lt;- evChannel
lg.Println(Csv(event))
}
}

this will give you a logger per goroutine.

In your version, you likely just had the last one to execute (probably last goroutine spawned, but not guaranteed)

And to improve a bit more, you probably also want your for loop written as:

for event := range evChannel {
lg.Println(Csv(event))
}

This way, it will terminate the goroutine when the channel is closed rather than spin on empty values coming out of a closed channel. See here for reference

huangapple
  • 本文由 发表于 2015年11月28日 13:40:02
  • 转载请务必保留本文链接:https://go.coder-hub.com/33968098.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定