Zipping large files(>15GB) and uploading to S3 without OOM

I have one memory issue while zipping large files/folders(result zip >15GB) and uploading it to S3 storage. I can create zip file in disc and append files/folders, upload that file with parts to S3. But by my experience it is not good way to resolve this issue. Do you know any good patterns zipping large files/folders and uploading it to S3 without memory issues(such OOM)? It will be good if i can append these files/folders to S3 directly to some uploaded zip.

Zip files/folders to disc and uploading that zip file by parts to S3.

You can use AWS Lambda to zip your files for you before uploading them to an S3 bucket. You can even configure Lambda to be triggered and zip your files on upload. Here is a Java example of a Lambda function for zipping large files. This library is limited to 10 GB, but this can be overcome by using EFS.

Lambda’s ephemeral storage is limited to 10 GB, but you can attach EFS storage to handle larger files. The cost should be close to none if you delete the files after use.

Also, remember to use Multipart Upload when uploading file larger than 100 MB to S3. If you are using the SDK, it should handle this for you.

👋

The main reason why you are getting an OOM is just because of how the deflate algorithm of zlib works.

Imagine this setup:

1. It starts to read the whole file by opening a readable stream.
2. It creates a temporary 0 byte output file from the start.
3. It then reads the data in chunks, called dictionary size, it then sends it to the CPU for further processing and compression, which are propagated back to the RAM.
4. When it finished with a certain fixed sized dictionary, it moves to the next one, and so on until it reaches END OF FILE terminator.
5. After that, it grabs all that deflated bytes (compressed) from RAM and writes that to the actual file.

You can observe & deduce that behavior by initiating a deflate operation, an example below.
(The file is created, 372mb is processed, but none is written to the file until the last processed byte.)

You could technically grab all of the parts, archive them AGAIN in a tar.gz and then upload to AWS, as one file, but you may get into the same problem with memory, but now on the uploading part.

Here are the file size limitations:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html

If you use the CLI you can technically do that, if you need or have to use the REST API that's not an option for you as the limitation there is only 5GB per request.

Also, you have not specified the max size, so if it's even larger than 160GB that's not an option EVEN using the AWS CLI (which takes care of releasing the memory after each uploaded chunk). So your best bet would be multipart upload.

https://docs.aws.amazon.com/cli/latest/reference/s3api/create-multipart-upload.html

All the best!

Zipping the file in 1 go is not exactly a correct way to go about. Think the better way is to break down the problem in a way you don't load the whole data in 1 go, but read it byte by byte and sent it to your destination byte by byte. This way, not only you will get speed (~x10) but also address those OOM's

Your destination could be a web end point on an EC2 instance or an API gateway fronted web service depending upon your architectural choice.

So essentially the part 1 of solution is to STREAM - zip it byte by byte and sent it to an http end point. Part 2 might be to use Multi part upload interfaces from AWS SDK (in your destination) and push it in parallel to S3

Path in = Paths.get("abc.huge");
Path out = Paths.get("abc.huge.gz");

try (InputStream in = Files.newInputStream(in);
    OutputStream fout = Files.newOutputStream(out);) {
	GZipCompressorOutputStream out2 = new GZipCompressorOutputStream(
   	 new BufferedOutputStream(fout));

	// Read and write byte by byte
 	final byte[] buffer = new byte[buffersize];
	 int n = 0;
	while (-1 != (n = in.read(buffer))) {
   	 out2.write(buffer, 0, n);
	}
}