How to merge 2 large pdf files in .Net Core without large memory utilization

When combining PDF files via code, every implementation I've seen, requires all files to be opened before being combined into one. This works just fine for small files, but with large enough files, we can exceed the resources available to an azure function, causing the function to restart.

I am wondering if there is a way to simply append one file to another from the outside, without having to open them. As an analogy, consider the this command that can be executed on a command line: "copy *.txt newfile.txt". It will take all files ending in .txt and create a new file that combines them all. If I understand correctly, I don't believe it requires loading the entire contents of any individual file into memory, as it is merely doing a file system copy, but joining the files where one file ends and the other begins. Is there a similar approach that can be taken with pdfs?

We've tried various merge implementations where you load the pages of each file into a for loop and add each page to a renderer, one by one. Obviously the memory usage is directly proportional to the size of the files in this approach.

Actually, this is possible using the incremental update feature, you can modify a PDF document without affecting its original content by appending just the changes at the end of the file.

Here is the Incremental Updates specification.

Also, here is an example of merging PDF files with incremental updates using GemBox.Pdf:
Merge a large number of PDF files

The main point is to use PdfDocument.Save() overload method that has no parameters and then PdfDocument.Unload() method. That way only one file will be loaded at any given time.

using (var document = new PdfDocument())
{
    document.Save("Merged Files.pdf");

    foreach (var file in files)
    {
        using (var source = PdfDocument.Load(file))
            document.Pages.Kids.AddClone(source.Pages);

        // Save the new pages.
        document.Save();

        // Clear previously parsed pages and thus free the memory.
        document.Unload();
    }
}

We can go even further with this and unload any parsed objects after X pages.

int counter = 0;
using (var destination = new PdfDocument())
{
    destination.Save("Merged Files.pdf");

    foreach (var file in files)
        using (var source = PdfDocument.Load(file))
            for (int index = 0, count = source.Pages.Count; index < count; counter++, index++)
            {
                destination.Pages.AddClone(source.Pages[index]);
                // Unload after 100 pages.
                if (counter % 100 == 0)
                {
                    destination.Save();
                    destination.Unload();
                    source.Unload();
                }
            }

    destination.Save();
}

TL;DR: Unfortunately, no.

Long version: Pdf is a fileformat that is structured as follows:

[File header]
PDF objects
[XRef Table]
[Trailer object]
[startxref offset]

The file header is a few simple bytes indicating that it is a PDF file, then some >128 bytes to indicate a binary file.
All PDF objects look like this:

1 0 obj
[object]
endobj

Where 1 is the object number. So object 2 would start as: 2 0 obj etc.

The XRef table is a list of objects and their byte offset in the file.

Objects can reference other objects. For example, the trailer object is a dictionary that contains a reference to the root element. The root element contains a reference to the pages object. The pages object contains one or more (might be an array) references to page object(s).

So when reading the PDF file, you start at the end, read the startxref value. That points to the start of the xref table. That table tells you at what byte offset the objects are, starting at object 1 0 obj.

So if you would concat 2 files together, when reading the file, you would read the startxref value. but the offset would be incorrect, because the offset is off by the length of the first file. Even when correcting for that. The entire xref table is off by the first file length. And even when correcting that, you can still only read the last PDF. To actually concat the pages, you need to edit all the object numbers (and the references in the file that use that object) to be [last object number from file 1 + n]. Then you need to overwrite the previous pages object to also include all the pages of the second file.

That is not the hardest, because you can just create a new PageTree object that will reference both pagetree object numbers. You do need to update/overwrite the Catalog object to reference the newly created pagetree as the new root of the tree.

Then you need to create a new xref table that uses all the objects from both files, list them in correct order and their byte offsets.