How can I connect two Scala Servers together to share a client pool using Akka w/ Play framework for Websockets

I have several HTTP servers with a RESTful API and a Scala WebSocket server utilizing Akka w/ Play Framework. The Scala Websocket (reactive) server is used by my mobile apps for realtime updates. The HTTP servers will communicate with the Scala server, to push data out to the mobile apps. This works great when there is 1 scala server, because all the mobile clients are connected to the same Scala server. However, I'd like to deploy a second websocket server, and this breaks the flow.

Scala 1

• Client 1
• Client 2

Scala 2

• Client 3
• Client 4

HTTP 1 (sends message for Client 3) -> LoadBalancer -> Scala 1

Scala 1 is not connected to Client 3, so the message never makes it to the client.

How can I connect the two Scala servers together, so even if the message is received by Scala 1, Scala 2 will send the message to Client 3? Or possibly share the client pool, so Scala 1 can send the message to the client over the socket, even if the Client initially connected to Scala 2?

I thought about implementing SNS and having the HTTP backend push events to a Queue, and then having the Scala backend subscribe to the Queue, but I'm not crazy about it because there are some events sent from the Mobile Client, that I also want both Scala servers to receive as well.
Example:
HTTP 1 (Sends Offer to All Clients) -> SNS -> Both Scala Servers Broadcast Offer to All Clients -> Client 3 Sends Accept to Scala -> Scala 2 receives, cancels offers to other clients on Scala 2, Scala 1 is unaware, keeps offer open to Clients connected on Scala 1

I'd have to publish the event to SNS again to cancel existing offers on the other server. It would be a lot of work.

I really need the two servers to act as one.

In general for this, what you will want to do is have the two servers form a cluster, which can effectively be thought of as allowing the two servers to form one system (for a variety of reasons, it's generally preferable to have at least three servers in the cluster). Then an HTTP or websocket connection to any of the servers can (transparently to whatever's on the other side of the HTTP/websocket connection) interact with any of the other servers.

The exact parts of the clustering toolkit in Akka to use will depend on the particulars. For instance, if most of the messages from HTTP are going to be intended for a single client and there's a small number (say less than a hundred thousand or so) of websocket clients and it's acceptable for a client to miss the occasional update, it's reasonably practical to use distributed pub sub within the cluster: when the client connects to the websocket, the handler spawns an actor which subscribes to a topic with a unique ID associated with the client. The handler for the HTTP request asking to publish a message to a given client will then publish a message to the topic associated with that client. If, conversely, your HTTP messages are mostly broadcasts and there are expected to be fewer sets-of-recipients than there are clients (and there are at most 10k or so sets-of-recipients), then you can invert this: have a topic for each set-of-recipients that's allowed and the clients subscribe to the appropriate topics with an HTTP request publishing to that topic.

If a guaranteed delivery of a message is required and you could have more than tens of thousands of clients and most messages will be point-to-point, then cluster sharding (and likely persistence, depending on how strong you want the guarantee to be around changes of cluster membership etc.) is a good tool: your HTTP request handler forwards a message for a given client to a sharded entity associated with that client, and your websocket handler essentially subscribes to that sharded entity (you'll likely have to implement the buffering, etc. functionality yourself: this is the sort of "more custom solution" the Akka docs talk about) and forwards acknowledgements from the client (such acknowledgements are intrinsically part of the domain problem): note that any given HTTP -message-> entity -message-> websocket -ack-> entity message flow could entail at least three or four network hops in the cluster (depending on whether you want to respond to the HTTP request with an indication of whether the message was definitely buffered for the websocket client or not).