英文:
Using multiple receivers for a channel
问题
我正在尝试使用go-json-rest在golang中编写一个REST服务。
该服务的目的只是将接收到的数据转换为CSV格式并记录下来。
由于负载可能很重,我希望使用goroutines进行日志记录。
目前我已经创建了四个LogWorker(goroutine)
每个goroutine将CSV记录到单独的文件中。
当我执行代码时,日志总是从最后一个goroutine触发。我在我的日志文件夹中只看到一个文件,来自第四个routine。
这是我的服务器代码:
package main
import (
"github.com/ant0ine/go-json-rest/rest"
"log"
"net/http"
"strconv"
"time"
)
const workerCount = 4
var evChannel = make(chan Event)
var workers = make([]*LogWorker, workerCount)
const maxLogFileSize = 100 // In MB
const maxLogFileBackups = 30
const maxLogFileAge = 5
const logFileName = "/home/sam/tmp/go_logs/event_"
func main() {
// 初始化 workers
// 创建了四个 workers
for i := 0; i < workerCount; i++ {
var fileName = logFileName + strconv.Itoa(i)
workers[i] = NewLogWorker(fileName, maxLogFileSize, maxLogFileBackups, maxLogFileAge)
go workers[i].Work(evChannel)
}
// 初始化 REST API
api := rest.NewApi()
//api.Use(rest.DefaultDevStack...)
api.Use(rest.DefaultCommonStack...)
router, err := rest.MakeRouter(
rest.Post("/events", StoreEvents),
)
if err != nil {
log.Fatal(err)
}
api.SetApp(router)
log.Fatal(http.ListenAndServe(":4545", api.MakeHandler()))
}
func StoreEvents(w rest.ResponseWriter, r *rest.Request) {
event := Event{}
err := r.DecodeJsonPayload(&event)
if err != nil {
rest.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// TODO : Add validation if needed
// Add code to parse the request and add further information to event
// log.Println()
select {
case evChannel <- event:
case <- time.After(5 * time.Second):
// throw away the message, so sad
}
// evChannel <- event
//log.Println(Csv(event))
w.WriteHeader(http.StatusOK)
}
这是我的worker代码:
package main
import (
"gopkg.in/natefinch/lumberjack.v2"
"log"
"fmt"
)
type LogWorker struct {
FileName string
MaxSize int // In megabytes
MaxBackups int // No of backups per worker
MaxAge int // maximum number of days to retain old log files
}
func NewLogWorker(fileName string, maxSize int, maxBackups int, maxAge int) (lw *LogWorker) {
return &LogWorker {fileName, maxSize, maxBackups, maxAge}
}
func (lw *LogWorker) Work(evChannel chan Event) {
fmt.Println(lw.FileName)
log.SetOutput(&lumberjack.Logger {
Filename: lw.FileName,
MaxSize: lw.MaxSize,
MaxBackups: lw.MaxBackups,
MaxAge: lw.MaxAge,
})
log.SetFlags(0)
for {
event := <- evChannel
log.Println(Csv(event))
}
}
请注意,event是一个包含一些字符串字段的结构体。
在SO上已经有一个类似的问题。当我尝试在playground中执行goroutine时,它仍然打印出最后一个goroutine的值。提供的答案中有一些wait.Done。由于我的worker需要持续运行,我认为我不能使用它。
请帮助我找出为什么我的所有goroutines(LogWorkers)都没有被使用?
英文:
I am trying write a REST service in golang using go-json-rest
The purpose of the service is just to convert the received data to CSV and log it.
Since the load may be heavy, I would like to do the logging using goroutines.
Currently I have created four LogWorkers(goroutine)
Each goroutine will log the CSV into separate files.
When I execute the code, the log is always triggered from the last goroutine. I see a single file created in my log folder which is from fourth routine.
Here is my server code
package main
import (
"github.com/ant0ine/go-json-rest/rest"
"log"
"net/http"
"strconv"
"time"
)
const workerCount = 4
var evChannel = make(chan Event)
var workers = make([]*LogWorker, workerCount)
const maxLogFileSize = 100 // In MB
const maxLogFileBackups = 30
const maxLogFileAge = 5
const logFileName = "/home/sam/tmp/go_logs/event_"
func main() {
// Initialize workers
// Four workers is being created
for i := 0; i < workerCount; i++ {
var fileName = logFileName + strconv.Itoa(i)
workers[i] = NewLogWorker(fileName, maxLogFileSize, maxLogFileBackups, maxLogFileAge)
go workers[i].Work(evChannel)
}
// Initialize REST API
api := rest.NewApi()
//api.Use(rest.DefaultDevStack...)
api.Use(rest.DefaultCommonStack...)
router, err := rest.MakeRouter(
rest.Post("/events", StoreEvents),
)
if err != nil {
log.Fatal(err)
}
api.SetApp(router)
log.Fatal(http.ListenAndServe(":4545", api.MakeHandler()))
}
func StoreEvents(w rest.ResponseWriter, r *rest.Request) {
event := Event{}
err := r.DecodeJsonPayload(&event)
if err != nil {
rest.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// TODO : Add validation if needed
// Add code to parse the request and add further information to event
// log.Println()
select {
case evChannel <- event:
case <- time.After(5 * time.Second):
// throw away the message, so sad
}
// evChannel <- event
//log.Println(Csv(event))
w.WriteHeader(http.StatusOK)
}
here is my worker code
package main
import (
"gopkg.in/natefinch/lumberjack.v2"
"log"
"fmt"
)
type LogWorker struct {
FileName string
MaxSize int // In megabytes
MaxBackups int // No of backups per worker
MaxAge int // maximum number of days to retain old log files
}
func NewLogWorker(fileName string, maxSize int, maxBackups int, maxAge int) (lw *LogWorker) {
return &LogWorker {fileName, maxSize, maxBackups, maxAge}
}
func (lw *LogWorker) Work(evChannel chan Event) {
fmt.Println(lw.FileName)
log.SetOutput(&lumberjack.Logger {
Filename: lw.FileName,
MaxSize: lw.MaxSize,
MaxBackups: lw.MaxBackups,
MaxAge: lw.MaxAge,
})
log.SetFlags(0)
for {
event := <- evChannel
log.Println(Csv(event))
}
}
Please note that event is a struct which contains some string fields.
Already there is a similar question in SO. When I tried to execute the goroutine in playground, it still prints the value from last go routine. The answer provided has some wait.Done. As my worker needs to run continuously I don't I think I can use it.
Please help me to find why my all goroutines (LogWorkers) are not used?
答案1
得分: 4
你正在为每个goroutine设置日志包的默认全局记录器的输出。
你可能想要做更多的事情,像这样:
func (lw *LogWorker) Work(evChannel chan Event) {
fmt.Println(lw.FileName)
lg := log.New(&lumberjack.Logger {
Filename: lw.FileName,
MaxSize: lw.MaxSize,
MaxBackups: lw.MaxBackups,
MaxAge: lw.MaxAge,
}, "", 0)
for {
event := <- evChannel
lg.Println(Csv(event))
}
}
这样将为每个goroutine提供一个记录器。
在你的版本中,你可能只有最后一个执行的记录器(可能是最后一个生成的goroutine,但不能保证)。
为了进一步改进,你可能还想将你的for循环写成:
for event := range evChannel {
lg.Println(Csv(event))
}
这样,当通道关闭时,它将终止goroutine,而不是在关闭的通道中旋转空值。参考这里
英文:
You are setting the log package's default global logger's output in each goroutine.
You probably want to do something more like:
func (lw *LogWorker) Work(evChannel chan Event) {
fmt.Println(lw.FileName)
lg := log.New(&lumberjack.Logger {
Filename: lw.FileName,
MaxSize: lw.MaxSize,
MaxBackups: lw.MaxBackups,
MaxAge: lw.MaxAge,
}, "", 0)
for {
event := <- evChannel
lg.Println(Csv(event))
}
}
this will give you a logger per goroutine.
In your version, you likely just had the last one to execute (probably last goroutine spawned, but not guaranteed)
And to improve a bit more, you probably also want your for loop written as:
for event := range evChannel {
lg.Println(Csv(event))
}
This way, it will terminate the goroutine when the channel is closed rather than spin on empty values coming out of a closed channel. See here for reference
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论