Get same size response as from first API fetch

So to first start, let's say I have an API which gives a ~300kB worth of data. Sometimes, the API is not responding for even 5 or 6 seconds (This is not our API, hence we cannot implement any logic inside it). Hence, we thought of creating another API which runs in NodeJS environment. The main idea is to refetch the data once in a while from the original API and store information inside REDIS database for better performance. In that way, we do not need to wait for stalled original API and fetch the data from our API.

However, the first fetch (from the original API to our API) is worth 300kB of data, but the fetch from our API to our project now becomes 1.5MB.

Is there any possibility to bring those number to the original size?

The original API server is probably doing some sort of compression using gzip or deflate on its response, and your fetch will automatically be decompressing it.

To get a similar size when requesting it you just need to compress it yourself after fetching it. You can either do this on the fly for each request, or store the compressed response in Redis (Probably the better of the 2 options).

If you are using Express you can use Express's compression middleware, which will compress on the fly, by just adding the below line.

app.use(compression())

which will automatically compress any responses over 1kb.

If you want to cache the compressed version on the Redis, the preferable method since it reduces memory usage on the Redis instance and also makes the request slightly faster you can use zlib

const zlib = require('zlib');

const apiResponse = '...'; // fetch from API, get content as string

zlib.gzip(apiResponse, (err, buffer) => {
    const valueToSave = buffer.toString('utf8');
    // Redis logic here.
})

And whenever you make the request to your API all you need to do is append the Content-Encoding: gzip header to tell the client how to decode the response.

Try enabling compression in your application. In ExpressJs, add below library:

npm i compression --save

Now to enable it in you app, use below in the main file:

// app.js
// Import the library
const compression = require('compression');
const express = require('express');
const app = express();

// Below line will compress all the HTTP responses
app.use(compression());

app.get('/', (req, res) => {
  res.send("<Your bulky data>");
});

app.listen(3000, function () {
  console.log('Started on port 3000!');
});