Golang concurrency access to slice

My use case:

• append items(small struct) to a slice in the main process
• every 100 items I want to process items in a processor go routine (then pop them from slice)
• items comme in very fast continuously

I read that if there is at least one "write" in more then two goroutines using a variable (slice in my case), one shall handle the concurrency (mutex or similar).

My questions:

• If I do not handle with a mutex the r/w on slice do I risk problems ? (ie. Item 101 arrives while the processor is working on 1-100s)
• What is the best concurrency technique for the incoming item flow to remain "fluent" ?

Disclaimer:

• I do not want any event queueing, I need to process items in a given "bundle" size

Actually you don't need a lock here. Here is a working code:

package main

import (
	"fmt"
	"sync"
)

type myStruct struct {
	Cpt int
}

func main() {
	buf := make([]myStruct, 0, 100)
	wg := sync.WaitGroup{}
	// Main process
	// Appending one million times
	for i := 0; i < 10e6; i++ {
		// Locking buffer
		// Appending
		buf = append(buf, myStruct{Cpt: i})
		// Did we reach 100 items ?
		if len(buf) >= 100 {
			// Yes we did. Creating a slice from the buffer
			processSlice := make([]myStruct, 100)
			copy(processSlice, buf[0:100])
			// Emptying buffer
			buf = buf[:0]
			// Running processor in parallel
			// Adding one element to waitgroup
			wg.Add(1)
			go processor(&wg, processSlice)
		}
	}
	// Waiting for all processors to finish
	wg.Wait()
}

func processor(wg *sync.WaitGroup, processSlice []myStruct) {
	// Removing one element to waitgroup when done
	defer wg.Done()
	// Doing some process
	fmt.Printf("Procesing items from %d to %d\n", processSlice[0].Cpt, processSlice[99].Cpt)
}

A few notes about your problem and this solution:

• If you want a minimal stop time in your feeding process (e.g, to respond as fast as possible to a HTTP call), then the minimal thing to do is just the copy part, and run the processor function in a go routine. By doing so, you have to create a unique process slice dynamically and copying the content of your buffer inside it.

• The sync.WaitGroup object is needed to ensure that the last processor function has ended before exiting the program.

Note that this is not a perfect solution: If you run this pattern for a long time, and the input data comes in more than 100 times faster than the processor processes the slices, then there are going to be:

• More and more processSlice instances in RAM -> Risks for filling the RAM and hitting the swap
• More and more parallel processor goroutines -> Same risks for the RAM, and more to process in the same time, making each of the calls be slower and the problem gets self-feeding.

This will end up in the system crashing at some point.

The solution for this is to have a limited number of workers that ensures there is no crash. However, when this number of workers is fully busy, then there will be wait in the feeding process, which does not answer what you want. However this is a good solution to absorb a charge which intensity is changing in time.

In general, just remember that if you feed more data than you can process in the same time, your program will just reach a limit at some point where it can't handle it so it has to slow down input acquisition or crash. This is mathematical!