How to separate array processing into goroutines?

I have a slice of 30'000 strings. How do I separate processing this slice into, say, 10 goroutines that would take 3000 strings from the slice, extract some data from them and push into a new slice?

So, in the end, I will have 10 slices with 3000 processed results in each. What's the pattern to handle this problem?

I have had a look at this article, but not sure which of these patterns applies to my case.

Using a channel, read the elements from the slice, use a Fan out to distribute load and pass messages. Then, process the strings in goroutines and collect the results back (fan in ) in a single goroutine to avoid mutexes.

You may want to set the number of Max concurrent concurrent goroutines.

Keep in mind that slices are not thread safe when writing to them.

Useful info:

> https://blog.golang.org/pipelines
> https://talks.golang.org/2012/concurrency.slide#1
> https://blog.golang.org/advanced-go-concurrency-patterns
> https://talks.golang.org/2013/advconc.slide#1

I agree with @JimB in that why limit the go routines. However since that's your inquiry I'd maybe so something like this.. If you really want to have each gorountine do 3000 items, then it might be easier to create a 2d slice. [[3000 items],[3000 items],..] then have 1 go routine process per index in that 2d array. Otherwise below just limits the gorountines to 10.
METHOD 1
package main

import (
"crypto/rand"
"fmt"
"log"
"sync"
"time"
)
var s []string
// genetate some mock data
func init() {
s = make([]string, 30000)
n := 5
for i := 0; i < 30000; i++ {
b := make([]byte, n)
if _, err := rand.Read(b); err != nil {
panic(err)
}
s[i] = fmt.Sprintf("%X", b)
}
}
func main() {
// set the number of workers
ch := make(chan string)
var mut sync.Mutex
counter := 0
// limit the number of gorountines to 10
for w := 0; w < 10; w++ {
go func(ch chan string, mut *sync.Mutex) {
for {
// get and update counter using mux to stop race condtions
mut.Lock()
i := counter
counter++
mut.Unlock()
// break the loop
if counter > len(s) {
return
}
// get string
myString := s[i]
// to some work then pass to channel
ch <- myString
}
}(ch, &mut)
}
// adding time.  If you play wiht the number of gorountines youll see how changing the number above can efficiency 
t := time.Now()
for i := 0; i < len(s); i++ {
result := <-ch
log.Println(time.Since(t), result, i)
}
}

METHOD2 the init function is creating a 2d array chunked out into 10 arrays each containing 3000 elements.. if you parse your data that way the logic below that needs very little modification to work

package main
import (
"crypto/rand"
"fmt"
"log"
"sync"
)
var chunkedSlice [10][3000]string
// genetate some mock data
// 2d array, each chunk has 3000 items in it
// there are 10 chunks, 1 go rountine per chunk
func init() {
n := 5
for i := 0; i < 10; i++ {
for j := 0; j < 3000; j++ {
b := make([]byte, n)
if _, err := rand.Read(b); err != nil {
panic(err)
}
chunkedSlice[i][j] = fmt.Sprintf("%X", b)
}
}
}
func main() {
// channel to send parsed data to
ch := make(chan string)
var wg sync.WaitGroup
// 10 chunks
for _, chunk := range chunkedSlice {
wg.Add(1)
// if defining the 2d array e.g [10][3000]string, you need to pass it as a pointer to avoid stack error
go func(ch chan string, wg *sync.WaitGroup, chunk *[3000]string) {
defer wg.Done()
for i := 0; i < len(chunk); i++ {
str := chunk[i]
// fmt.Println(str)
// parse the data (emulating)
parsedData := str
// send parsed data to the channel
ch <- parsedData
}
}(ch, &wg, &chunk)
}
// wait for all the routines to finish and close the channel
go func() {
wg.Wait()
close(ch)
}()
var counter int // adding to check that the right number of items was parsed
// get the data from the channel
for res := range ch {
log.Println(res, counter)
counter++
}
}

I have developed a library parapipe to solve such task. Just slice your slice into 10 parts and send them to Pipeline, which will process them concurrently:

import "github.com/nazar256/parapipe"
//...
var longStringSlice []string
// ...
pipeline := parapipe.NewPipeline(10).
Pipe(func(msg interface{}) interface{} {
slicePart := msg.([]string)
// process here to result
return result
})
// chop the slice and stream parts
chopSize := int(math.Ceil(float64(len(longStringSlice)) / 10))
for i:=0;i<10;i++ {
firstIdx := i * chopSize
lastIdx := (i+1) * chopSize
if lastIdx > len(longStringSlice) {
lastIdx = len(longStringSlice)
}
pipeline.In() <- longStringSlice[firstIdx:lastIdx]
}

There is no problem handling different memory addresses in different goroutines. If your read and write actions do not overshadow the chunks, that you can simply work with ranges in massif.
https://play.golang.org/p/MU5njoNmIf7