Parallel zip compression in Go

I am trying build a zip archive from a large number of small-medium sized files. I want to be able to do this concurrently, since compression is CPU intensive, and I'm running on a multi core server. Also I don't want to have the whole archive in memory, since its might turn out to be large.

My question is that do I have to compress every file and then combine manually combine everything together with zip header, checksum etc?

Any help would be greatly appreciated.

I don't think you can combine the zip headers.

What you could do is, run the zip.Writer sequentially, in a separate goroutine, and then spawn a new goroutine for each file that you want to read, and pipe those to the goroutine that is zipping them.

This should reduce the IO overhead that you get by reading the files sequentially, although it probably won't leverage multiple cores for the archiving itself.

Here's a working example. Note that, to keep things simple,

• it does not handle errors nicely, just panics if something goes wrong,
• and it does not use the defer statement too much, to demonstrate the order in which things should happen.

Since defer is LIFO, it can sometimes be confusing when you stack a lot of them together.

package main
import (
"archive/zip"
"io"
"os"
"sync"
)
func ZipWriter(files chan *os.File) *sync.WaitGroup {
f, err := os.Create("out.zip")
if err != nil {
panic(err)
}
var wg sync.WaitGroup
wg.Add(1)
zw := zip.NewWriter(f)
go func() {
// Note the order (LIFO):
defer wg.Done() // 2. signal that we're done
defer f.Close() // 1. close the file
var err error
var fw io.Writer
for f := range files {
// Loop until channel is closed.
if fw, err = zw.Create(f.Name()); err != nil {
panic(err)
}
io.Copy(fw, f)
if err = f.Close(); err != nil {
panic(err)
}
}
// The zip writer must be closed *before* f.Close() is called!
if err = zw.Close(); err != nil {
panic(err)
}
}()
return &wg
}
func main() {
files := make(chan *os.File)
wait := ZipWriter(files)
// Send all files to the zip writer.
var wg sync.WaitGroup
wg.Add(len(os.Args)-1)
for i, name := range os.Args {
if i == 0 {
continue
}
// Read each file in parallel:
go func(name string) {
defer wg.Done()
f, err := os.Open(name)
if err != nil {
panic(err)
}
files <- f
}(name)
}
wg.Wait()
// Once we're done sending the files, we can close the channel.
close(files)
// This will cause ZipWriter to break out of the loop, close the file,
// and unblock the next mutex:
wait.Wait()
}

Usage: go run example.go /path/to/*.log.

This is the order in which things should be happening:

1. Open output file for writing.
2. Create a zip.Writer with that file.
3. Kick off a goroutine listening for files on a channel.
4. Go through each file, this can be done in one goroutine per file.
5. Send each file to the goroutine created in step 3.
6. After processing each file in said goroutine, close the file to free up resources.
7. Once each file has been sent to said goroutine, close the channel.
8. Wait until the zipping has been done (which is done sequentially).
9. Once zipping is done (channel exhausted), the zip writer should be closed.
10. Only when the zip writer is closed, should the output file be closed.
11. Finally everything is closed, so close the sync.WaitGroup to tell the calling function that we're good to go. (A channel could also be used here, but sync.WaitGroup seems more elegant.)
12. When you get the signal from the zip writer that everything is properly closed, you can exit from main and terminate nicely.

This might not answer your question, but I've been using similar code to generate zip archives on-the-fly for a web service some time ago. It performed quite well, even though the actual zipping was done in a single goroutine. Overcoming the IO bottleneck can already be an improvement.

From the look of it, you won't be able to parallelise the compression using the standard library archive/zip package because:

1. Compression is performed by the io.Writer returned by zip.Writer.Create or CreateHeader.
2. Calling Create/CreateHeader implicitly closes the writer returned by the previous call.

So passing the writers returned by Create to multiple goroutines and writing to them in parallel will not work.

If you wanted to write your own parallel zip writer, you'd probably want to structure it something like this:

1. Have multiple goroutines compress files using the compress/flate module, and keep track of the CRC32 value and length of the uncompressed data. The output should be directed to temporary files. Note the compressed size of the data.
2. Once everything has been compressed, start writing the Zip file starting with the header.
3. Write out the file header followed by the contents of the corresponding temporary file for each compressed file.
4. Write out the central directory record and end record at the end of the file. All the required information should be available at this point.

For added parallelism, step 1 could be performed in parallel with the remaining steps by using a channel to indicate when compression of each file completes.

Due to the file format, you won't be able to perform parallel compression without either storing compressed data in memory or in temporary files.

With Go1.17, parallel compression and merging of zip files are possible using the archive/zip package.

An example is below. In the example, I create zip workers to create individual zip files and an entry provider worker which provides entries to be added to a zip file via a channel to zip workers. Actual files can be provided to the zip workers but I skipped that part.

package main

import (
	"archive/zip"
	"context"
	"fmt"
	"io"
	"log"
	"os"
	"strings"

	"golang.org/x/sync/errgroup"
)

const numOfZipWorkers = 10

type entry struct {
	name string
	rc   io.ReadCloser
}

func main() {
	log.SetFlags(log.LstdFlags | log.Lshortfile)

	entCh := make(chan entry, numOfZipWorkers)
	zpathCh := make(chan string, numOfZipWorkers)

	group, ctx := errgroup.WithContext(context.Background())

	for i := 0; i < numOfZipWorkers; i++ {
		group.Go(func() error {
			return zipWorker(ctx, entCh, zpathCh)
		})
	}

	group.Go(func() error {
		defer close(entCh) // Signal workers to stop.
		return entryProvider(ctx, entCh)
	})

	err := group.Wait()
	if err != nil {
		log.Fatal(err)
	}

	f, err := os.OpenFile("output.zip", os.O_CREATE|os.O_TRUNC|os.O_WRONLY, 0644)
	if err != nil {
		log.Fatal(err)
	}

	zw := zip.NewWriter(f)

	close(zpathCh)
	for path := range zpathCh {
		zrd, err := zip.OpenReader(path)
		if err != nil {
			log.Fatal(err)
		}
		for _, zf := range zrd.File {
			err := zw.Copy(zf)
			if err != nil {
				log.Fatal(err)
			}
		}
		_ = zrd.Close()
		_ = os.Remove(path)
	}
	err = zw.Close()
	if err != nil {
		log.Fatal(err)
	}
	err = f.Close()
	if err != nil {
		log.Fatal(err)
	}
}

func entryProvider(ctx context.Context, entCh chan<- entry) error {
	for i := 0; i < 2*numOfZipWorkers; i++ {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case entCh <- entry{
			name: fmt.Sprintf("file_%d", i+1),
			rc:   io.NopCloser(strings.NewReader(fmt.Sprintf("content %d", i+1))),
		}:
		}
	}
	return nil
}

func zipWorker(ctx context.Context, entCh <-chan entry, zpathch chan<- string) error {
	f, err := os.CreateTemp(".", "tmp-part-*")
	if err != nil {
		return err
	}

	zw := zip.NewWriter(f)
Loop:
	for {
		var (
			ent entry
			ok  bool
		)
		select {
		case <-ctx.Done():
			err = ctx.Err()
			break Loop
		case ent, ok = <-entCh:
			if !ok {
				break Loop
			}
		}

		hdr := &zip.FileHeader{
			Name:   ent.name,
			Method: zip.Deflate, // zip.Store can also be used.
		}
		hdr.SetMode(0644)

		w, e := zw.CreateHeader(hdr)
		if e != nil {
			_ = ent.rc.Close()
			err = e
			break
		}

		_, e = io.Copy(w, ent.rc)
		_ = ent.rc.Close()
		if e != nil {
			err = e
			break
		}
	}

	if e := zw.Close(); e != nil && err == nil {
		err = e
	}
	if e := f.Close(); e != nil && err == nil {
		err = e
	}
	if err == nil {
		select {
		case <-ctx.Done():
			err = ctx.Err()
		case zpathch <- f.Name():
		}
	}
	return err
}