TCPStream random wait and retry if server is not listening

My client code is like this:

use tokio::net::TcpStream;
use tokio::io::AsyncWriteExt;
use std::error::Error;
use socket2;
use std::{ time};

    #[tokio::main]
    pub async fn match_tcp_client(address: String, self_ip: String) -> Result<(), Box<dyn Error>> {
        // Connect to a peer
        println!("trying to connect from {} to address {}", self_ip, address);
    
        let mut stream = TcpStream::connect(address.clone()).await?;
    
    
        println!("connected from {} to address {}", self_ip, address);
    
    
        let sock_ref = socket2::SockRef::from(&stream);
    
        let mut ka = socket2::TcpKeepalive::new();
        ka = ka.with_time(time::Duration::from_secs(20));
        ka = ka.with_interval(time::Duration::from_secs(20));
    
        sock_ref.set_tcp_keepalive(&ka)?;
    
    
        // Write some data.
        stream.write_all(self_ip.as_bytes()).await?;
        stream.write_all(b"hello world!EOF").await?;
        
        
        Ok(())
    }

I want client to wait a random time and then retry if server if not active (and listening ) yet.
Please note, the TcpKeepalive I tried didnt work.

Is there something like a loop where I can check if server is alive, if not wait a random time and try again?

Something I tried:

  use tokio::net::TcpStream;
    use tokio::io::AsyncWriteExt;
    use std::error::Error;
    use socket2;
    use std::{ time};
    use tokio::time::{ sleep, Duration};
    
    #[tokio::main]
    pub async fn match_tcp_client(address: String, self_ip: String) -> Result<(), Box<dyn Error>> {
        // Connect to a peer
        println!("trying to connect from {} to address {}", self_ip, address);
    
        while TcpStream::connect(address.clone()).await.is_err()
        {
            sleep(Duration::from_millis(10)).await;
        }
    
        let mut stream = TcpStream::connect(address.clone()).await?;
    
    
        println!("connected from {} to address {}", self_ip, address);
    
    
        let sock_ref = socket2::SockRef::from(&stream);
    
        let mut ka = socket2::TcpKeepalive::new();
        ka = ka.with_time(time::Duration::from_secs(20));
        ka = ka.with_interval(time::Duration::from_secs(20));
    
        sock_ref.set_tcp_keepalive(&ka)?;
    
    
        // Write some data.
        stream.write_all(self_ip.as_bytes()).await?;
        stream.write_all(b"hello world!EOF").await?;
        
        
        Ok(())
    }

But it is not working as well. In this case, though for all EC2 instances (I am using 4 instance), I am getting the log:

connected from A to address A
trying to connect from A to address B
connected from A to address B
trying to connect from A to address C
connected from A to address C
trying to connect from A to address D
connected from A to address D

But then for some reason the stream writing is not starting and the instances remain paused.

Edit:
I also tried this (same result as before):

 use tokio::net::TcpStream;
    use tokio::io::AsyncWriteExt;
    use std::error::Error;
    use socket2;
    use std::{ time};
    use tokio::time::{ sleep, Duration};
    
    #[tokio::main]
    pub async fn match_tcp_client(address: String, self_ip: String) -> Result<(), Box<dyn Error>> {
        // Connect to a peer
        println!("trying to connect from {} to address {}", self_ip, address);
    
        
            loop{
        if TcpStream::connect(address.clone()).await.is_ok(){
        let mut stream = TcpStream::connect(address.clone()).await?;
        stream.set_linger(None)?;
    
        println!("connected from {} to address {}", self_ip, address);
    
    
        let sock_ref = socket2::SockRef::from(&stream);
    
        let mut ka = socket2::TcpKeepalive::new();
        ka = ka.with_time(time::Duration::from_secs(20));
        ka = ka.with_interval(time::Duration::from_secs(20));
    
        sock_ref.set_tcp_keepalive(&ka)?;
    
    
        // Write some data.
        stream.write_all(self_ip.as_bytes()).await?;
        stream.write_all(b"hello world!EOF").await?;
        
        stream.shutdown().await?;
        break;
        }
    
    }
        Ok(())
    }

You are trying to connect to the address then dropping the connection if it was established and try to connect again. That is is not how to do it. Instead start a loop and try to connect and if successful return an Option from the loop containing Some(stream) otherwise wait before retrying and if you exceeded the maximum try times return a None from the loop. Although I think Result will be more favourable here.

Another thing is that you are dropping the stream in the end of the function after awaiting write thinking that the data was already sent! When write method completes this means that the OS has copied the supplied buffers into the kernel space and it no longer needs the user buffers. The kernel then starts writing the data in chunks and failed writes are retried until success pr connection is considered broken.

Now when you drop the stream will the OS continue sending the data in the background? This depends on lingering and whether it is enabled or not. Different OSes and libraries have different default settings. Coming from c++ asio networking library lingering is disabled by default and if the stream is dropped cached data will not be sent.

To force the kernel to send your data either use setsocketopt to enable lingering if tokio allows you to do this.

Or better use shutdown which will always force the kernel to flush all accumulated data on the socket and thus ensure that all data are transferred if the connection is still active.

Edit: tokio allows you to configure linger: https://docs.rs/tokio/latest/tokio/net/struct.TcpStream.html#method.set_linger

Also there is shutdown here: https://docs.rs/tokio/latest/tokio/io/trait.AsyncWriteExt.html#method.shutdown