Serve response of an HTTP request after receiving another request

My use case is to serve response of an HTTP request after receiving another request from separate server.

1. I want to do this best possible way keeping scaling in mind.
2. We are using Golang 1.19 with Gin Framework.
3. Server will have multiple pods thus channels will not work.
4. There will be timeouts for all request making initial request timed out after 60 seconds.

My current solution is to use a shared cache where each pod will keep checking the cache. I believe, I can optimize this with channels where rather than checking in cache one by one, system periodically checks for any completed response.

I would also like to know how it could have been achieved in other programming languages.

PS: This is design based query, I have some reputation here to share bounty thus asking here. Please feel free to edit if question is not clear.

• client fires HTTP call to server A.
• server A creates a unique key & store it in cache server for 60s(timeout period).
• server A forwards the request to B via HTTP call. It is a fire & forget call to B.
• server A immediately returns response to the client with the unique key.
• client starts polling (let's say every 500ms) using a GET status HTTP API to check if processing is done at server A.
• mean while server B completes the task & calls back to server A via HTTP API.
• server A stores the response in cache against the unique key for short period (let's say 60s).
• client call to A for status check API will get the data from cache & returns it to the client.

There is no other component except server A & server B. There is cache server, but it is internal to A. B doesn't need to know about it. Which service maintains what is clearly defined, so easy to maintain. B can have internal queue to process the forwarded request. But A doesn't need to know how B is implemented. Each service can be maintained by different teams using simple HTTP contract.

Another advantage is that client is not holding a long running HTTP connection to A. I mentioned polling thinking of most primitive clients like an old browser. If your client supports web socket, you can use that to send response back from A. Anyways polling here would perform well as status check API is just a cache call & nothing else.

Someone can ask, where is the retry logic for server to server communication between A & B? But we don't need a queue for that. Essentially client wants a synchronous type of call here. We are just breaking it down to multiple calls. Response anyways needs to be quick. We can have a retry mechanism of 3-5 retries in the HTTP client in case of network failure. We are using pods. I expect it to be behind a k8s load balancer. That load balancer will anyway retry to a healthy pod if first pod goes down. So you are pretty much covered there.

I don't know your exact requirements & I am writing this after a quick thought. But overall this looks fine to me. It would be robust & have low latency calls. Maybe few tweaks would be needed here & there.

Update based on comment:

Implementation of client polling is pretty basic. Basically, it is a loop which runs every 0.5s & makes HTTP GET call to check the status. I believe any client should be able to implement that. But there might be some restrictions where client team don't want to make any changes. I don't want to get into that.

Let's say some clients don't have polling capabilities & some clients don't have web socket support. Then only thing remaining is a long lived HTTP connection between client & server A. Suppose we take this approach. Not going into the obvious problems of having a long lived request thread in server, there is another issue specific to this scenario. Request thread in server A needs to be notified when server B makes the callback.

Approach 1: We use some kind of queue as mentioned the other answer (I would keep the queue internal to A, but let's not get into that). And we use some message key so that the consumption happens in the same pod. We have API & consumer both running in the same pod. That is achievable. But request thread & consumer thread are different. Consumer thread somehow needs to notify request thread that result is available. So some inter-thread communication is required which only makes things complex. And it is not responsibility of a queue. So I would discard this approach.

Someone can ask, can request thread directly listens for some message with specific key. I don't think so. For the argument's sake, there are some queue technologies where you can do that. But you are essentially having ephemeral consumer & separate partition for each key which won't scale. Listening for all messages & ignoring all but one in the ephemeral consumer is not a viable solution either.

Approach 2: From approach 1, we understood that request thread needs to be notified when server B sends the callback. That would make the flow whole lot complex & you would need additional components like ZooKeeper to do the distributed locking or watching on some change. Instead of doing that, we can simply extend our current system to do server side polling. It shouldn't have any noticeable difference in response time as we are already doing client side polling. We are just moving it to server side. Also, anyways we would have to maintain long running request thread in server A whether we built distributed notification or server polling. The flow would look like:

• Client calls server A.
• Server A request thread makes a HTTP call to server B.
• Server B starts background processing & returns HTTP response immediately to A.
• Request thread in A starts a loop checking in cache server if key is available every 500ms.
• B processes the result & makes HTTP callback to A.
• callback thread in A caches the result against same key.
• Original A request thread reads the value from cache & returns the response to client.

Original request thread would be sleeping for 500ms before making next cache call. So other threads would be able to utilize the time. Also cache get by key is super fast & you won't have any scalability issue there.

But you will exhaust your request threads quicker if you maintain long running HTTP connection from client. So you will need more A pods to handle same request rate. I would still recommend to talk to the client team & make the necessary changes there so that short lived connection (same as your current flow) is used.

tl;dr

problem description

So assuming your server application called server_app for instance, has 3 pods:

     +---------------------+
     |  server_app_service |
     +---------------------+
     |  server_app_pod_a   |
     |  server_app_pod_b   |
     |  server_app_pod_c   |
     +---------------------+

your service receives a request called "request A", and decides to pass it to server_app_pod_a. Now your server_app_pod_a forwards the request to some gateway, and waits for some sort of notification, to continue the processing of client's response. And as you already know, there's no assurance that when gateway does the request B, the service passes it to server_app_pod_a again. And even if it did so, your application's state management would become a difficult task to do.

Messaging

As you might've noticed, I bolded the word "notification" in the past paragraph, that's because if you really think about it, the request "B" looks more like a notification with some message rather than a request for some resource. So my number 1 choice would be a message queue like kafka (there are plenty of those, again, as you know). And the idea is, if you could define an algorithm to calculate unique keys for your requests, you can expect the resulting notifications in your exact same pod. This way, state management would be much simpler, and also, the chance of getting the notification in the same pod would be much higher (this depends on many factors of course, like the state of the message queue). Taking a look at your questions:

> 1. I want to do this best possible way keeping scaling in mind.

Sure thing, you can use these message queues like kafka, to achieve scaling and fewer data loss, both for the message queue and your application.

> 4. There will be timeouts for all request making initial request timed out after 60 seconds.

This one depends on how you manage timeouts in your codebase, using contexts would be a good idea.

> I would also like to know how it could have been achieved in other programming languages.

Using message queues is a general idea, which would be applicable to almost any programming language, but depending on the programming paradigms of a language, and language-specific libraries and tools, there might be some other approaches to this problem. For instance in Scala, if you use some specific tool called akka (which provides actor model programming paradigm), you can use something so called akka-cluster-sharding, to handle this problem. And the idea is pretty simple, we know that there must be some sort of superviser, which knows the exact location and state of its own subscribers. So when it receives some message, it just knows where and which actor (we're talking about actor model programming) to forward the request to. In other words, it can be used to share state between actors spawned on a cluster, either on the same machine or not. But as a personal preference, I wouldn't go for language-specific communications, and would stick to general ideas, because of the problems it might cause in the future.

Wrap-up

Long enough explanations :). Just to make some sense of what I'm talking about, let's follow up the exact same scenario, with a difference in communication model:

1. Client sends request "A" to server_app service.
2. The service, choses one of the pods (server_app_pod_b for instance) to handle the request.
3. The pod then tries to define some key for the request, and passes it to the gateway, along with the request, and waits for a message with the key, to be published in the queue.
4. The gateway does what it's supposed to, and sends a message with the key, to the message queue.
5. The exact same pod serer_app_pod_b receives the message with the key, fetches the data of the message, and continues to process the client's request.

There are probably other approaches available to address this issue, but this is what I would go for. Hope that it helps!