When is it valid to call inflateSetDictionary() when trying to decompress raw deflate data that was compressed with a dictionary?

The Problem

When is it valid to call inflateSetDictionary() when trying to decompress raw deflate data that was compressed with a compression dictionary?

According to the zlib manual, it is stated that inflateSetDictionary() can be called "at any time". However, it is unclear to me what "at any time" actually means. If we are allowed to call inflateSetDictionary() "at any time", then I interpret it as being valid to call inflateSetDictionary() after calling inflate(). However, doing so results in inflate() returning an "invalid distance too far back" error.

My Code

I created a simple application to compress the string "hello" using raw deflate, with a compression dictionary that also consists of the byte sequence "hello":

#define BUF_SIZE 16384
#define WINDOW_BITS -15 // Negative for raw.
#define MEM_LEVEL 8

const unsigned char dictionary[] = "hello";

unsigned char uncompressed[BUF_SIZE] = "hello";
unsigned char compressed[BUF_SIZE];

z_stream deflate_stream;

deflate_stream.zalloc = Z_NULL;
deflate_stream.zfree = Z_NULL;
deflate_stream.opaque = Z_NULL;

deflateInit2(&deflate_stream,
             Z_DEFAULT_COMPRESSION,
             Z_DEFLATED,
             WINDOW_BITS,
             MEM_LEVEL,
             Z_DEFAULT_STRATEGY);

deflateSetDictionary(&deflate_stream, dictionary, sizeof(dictionary));

deflate_stream.avail_in = (uInt)strlen(uncompressed) + 1;
deflate_stream.next_in = (Bytef *)uncompressed;

deflate_stream.avail_out = BUF_SIZE;
deflate_stream.next_out = (Bytef *)compressed;

deflate(&deflate_stream, Z_FINISH);

deflateEnd(&deflate_stream);

This produced 4 bytes of raw deflate data into the compressed buffer:

uLong compressed_size = deflate_stream.total_out;
printf("Compressed size is: %lu\n", compressed_size); // prints Compressed size is: 4

I then attempted to decompress this data back into the string "hello". The zlib manual states that I would need to use raw inflate to decompress raw deflate data:

unsigned char decompressed[BUF_SIZE];

z_stream inflate_stream;

inflate_stream.zalloc = Z_NULL;
inflate_stream.zfree = Z_NULL;
inflate_stream.opaque = Z_NULL;

inflateInit2(&inflate_stream, WINDOW_BITS);

inflate_stream.avail_in = (uInt)compressed_size;
inflate_stream.next_in = (Bytef *)compressed;

inflate_stream.avail_out = BUF_SIZE;
inflate_stream.next_out = (Bytef *)decompressed;

int r = inflate(&inflate_stream, Z_FINISH);

According to the zlib manual, I would expect that inflate() should return Z_NEED_DICT, and I would then call inflateSetDictionary() with a subsequent call to inflate():

// Must be called immediately after a call of inflate, if that call returned Z_NEED_DICT.
if (r == Z_NEED_DICT) {
    inflateSetDictionary(&inflate_stream, dictionary, sizeof(dictionary));
    r = inflate(&inflate_stream, Z_FINISH);
}

if (r != Z_STREAM_END) {
    printf("inflate: %s\n", inflate_stream.msg);
    return 1;
}

inflateEnd(&inflate_stream);

printf("Decompressed size is: %lu\n", strlen(decompressed));
printf("Decompressed string is: %s\n", decompressed);

However, what ends up happening is that inflate() will not return Z_NEED_DICT, and instead return Z_DATA_ERROR, with the value of inflate_stream.msg being set to "invalid distance too far back".

Even if I were to adjust my code so that inflateSetDictionary() is called regardless of the return value of inflate(), the subsequent inflate() call will still fail with Z_DATA_ERROR due to "invalid distance too far back".

My Question

So far, my code works correctly if I were to use the default zlib encoding by setting WINDOW_BITS to 15, as opposed to -15 for the raw encoding.

My code also works correctly if I were to move the call for inflateSetDictionary() before the call to inflate().

However, it's not clear to me why my existing code does not allow inflate() to return Z_NEED_DICT, so that I can make a subsequent call to inflateSetDictionary().

Is there a mistake in my code somewhere that is preventing inflate() from returning Z_NEED_DICT? Or can inflateSetDictionary() only be called prior to inflate() for the raw encoding, contrary to what the zlib manual states?

inflate() will only return Z_NEED_DICT for a zlib stream, where the need for a dictionary is indicated by a bit in the zlib header, followed by the Adler-32 of the dictionary that was used for compression to verify or select the dictionary. There is no such indication in a raw deflate stream. There is no way for inflate() to know from a raw deflate stream whether or not the data was compressed with a dictionary. It is up to you to know what is needed for decompression, since you made the raw deflate stream in the first place.

Since you did a deflateSetDictionary() before compressing anything, it is up to you to do an inflateSetDictionary() at the same place, before you decompress, after the inflateInit(). As you have found, you need to insert:

    inflateSetDictionary(&inflate_stream, dictionary, sizeof(dictionary));

right after the inflateInit(). Then decompression will be successful.

Yes, you can do inflateSetDictionary() at any point during a raw deflate decompression. However it will only work if you are doing it at the same point at which you did the corresponding deflateSetDictionary() when compressing.