Which go concurrency pattern should I use for the following use case?

I'm having two go routines.

• NewFileEntryCheck()
• UploadFromQueueToCloud()

go NewFileEntryCheck()

This function will take a fsnotify.Watcher() as argument and it will watch for any file change in the specified directory inside an infinite loop.

If there is any new file CREATE event occurs, it will add the filename to the Queue

go UploadFromQueueToCloud()

This go routine should take the filename from the Queue read the file from the path and upload it to cloud.

Now, my question is, How can I make the UploadFromQueueToCloud() function to get value from the Queue and upload it to cloud?

Since the NewFileEntryCheck() function will continuously check for new files and add it to the queue, the length of the Queue is not fixed. I can't loop through the len(Queue) and receive data through the channel.

Is there any other approach to achieve this?

This is just a common example of pub-sub model. You have one publisher (NewFileEntryCheck) and one subscriber (UploadFromQueueToCloud). Because this can be modelled in the form of Pub-Sub model, you should make the design similarly.

There is already a comment that you might have received from @jimb asking that if the Queue is a channel. Well there is a reason for it. It's because golang channel behaves just like a queue. So one solution is definitely to model it using a channel.

Using channel, you can put the task/file name from one go routine. From another go routine UploadFromQueueToCloud, you can consume that channel and process the data asynchronously.

Here is the sample code

func UploadFromQueueToCloud(channel *chan string, exit *chan bool) {
   	for {
		select {
           case file := <- channel:
               // Upload the cloud..
               // Other business logics
           case exit := <- exit:
               break;
        }
	}
}

func NewFileEntryCheck(notifier *fsnotify.Watcher) {
   channel := make(chan string)
   exit := make(chan bool)
   for {
        if (some breaking condition) {
             exit <- true;
             break;
        }
        // Check for the file notifier
        // If file CREATE happened,
        // get the file name into FileName
        channel <- FileName;
        go UploadFromQueueToCloud(channel, exit)
   }
} 

Note: You can increase the throughput by changing the channel into a buffered channel.

Hope that helps.

My case is:

func main() {
	allFileCheck := make(chan bool)
	fileUpload := make(chan string)
	ctx, cancel := context.WithCancel(context.Background())
	go NewFileEntryCheck(ctx, fileUpload, allFileCheck)

L:
	for {
		select {
		case file := <- fileUpload:
			go UploadFromQueueToCloud(ctx, file)
		case <-allFileCheck:
			fmt.Println("done")
			break L
		case <-time.After(5 * time.Second):
			cancel()
		}
	}
}

func UploadFromQueueToCloud(ctx context.Context, file string) {
	fmt.Printf("Upload file %v\n", file)
}

func NewFileEntryCheck(ctx context.Context, fileUpload chan string, allFileCheck chan bool) {
	fmt.Println("Check file")
	for i := 0; i < 5; i++ {
		fileUpload <- "name " + strconv.Itoa(i)
	}
	allFileCheck <- true
}