understanding go channel handling / buffer overflow

I have inherited some code I am still trying to make sense of. The core of it is this:

for msg := range this.out {
    for i, handler := range this.handlers {
        select {
        case handler <- msg:
        default:
            this.logger.Printf("Buffer overflow occurred for handler %s", this.names[i])
        }
    }
}

out is a chan byte
handlers is []chan []byte

It looks like this is reading from out and writing to a handler from the array, and the default is to complain about a buffer overflow. I think.

But I need details. I am new to go and this is my first time dealing with chan. So first question, is that actually what is happening here? And if so ... how do I prevent the buffer overflow?

In Go, channels can be created in two ways:

c := make(chan []byte)

which creates a channel that will synchronize the sender and receiver. This means that the receiver will have to wait until data is sent by the sender, and the sender will have to wait until the receiver gets that data. However, by adding a buffer to the channel:

c := make(chan []byte, 100)

you're effectively desynchronizing the sender and receiver. In this case, the sender will block when the buffer is full and the receiver will block when the buffer is empty.

Now, I don't know what the capacity on handler is, but this is the general workflow you should expect:

1. Go will check if there is space in handler and if there is, msg will be written to the channel.
2. If there is not space then the error message will be logged.

What you actually do about buffer overflow is a little more tricky.

First, as a stopgap, you could increase the capacity on handler so that it can hold more messages. This may represent an actual fix if the problem is intermittent (i.e. large spikes in traffic). Otherwise, all you're doing is pushing the issue off for a time.

Second, you could horizontally scale the handlers on the other end of handler to process more messages, thereby ensuring that the channel doesn't overflow. The problem with this, however, is that you then have to worry about managing handlers at various capacity levels so then you get into auto-scaling and things like that.

Third, you could look at the code handling messages from handler and see if this could be redesigned to handle messages more efficiently. Such redesigns are pretty common when the original design assumptions for your code no longer hold.

Finally, you could move away from using channels as infrastructure altogether and replace that component with a cloud-based pub/sub that includes memory. The advantage here is that this would allow you to scale your application as necessary without worrying about overflow but it would incur additional costs and require infrastructure changes.