One server block for domain, another server block for IP of same server

I'm using nodejs with nginx as my reverse proxy. In my nginx config I have a server block that is targeted when a client accesses my website via domain name:

server {
     root /var/www/html;
     index index.html index.htm index.nginx-debian.html;
     server_name example.com, www.example.com;

     location / {
            proxy_pass http://localhost:6000;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

 listen [::]:443 ssl http2 ipv6only=on;
 listen 443 ssl http2;
 ssl_certificate /etc/letsencrypt/live/www.example.com/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/www.example.com/privkey.pem;
 include /etc/letsencrypt/options-ssl-nginx.conf;
 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}

server {
     if ($host = www.example.com) {
          return 301 https://$host$request_uri;
     }  


    listen 80 default_server;
    listen [::]:80 default_server;

    server_name example.com, www.example.com;
    return 404;

}

So the above config works fine. But sometimes I need to hit an endpoint on my remote nodejs server by connecting to that server via IP address, and I want to do it securely. So I made a self signed cert on the remote server, and I added another server block like so:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name 149.154.150.134;

    ssl_certificate /path/to/cert/fullchain.pem;
    ssl_certificate_key /path/to/cert/privkey.pem;

    location / {
            allow 214.107.49.66;
            deny all;
    }

}

But nginx always seems to issue the let's encrypt cert, even when I try accessing the server via IP address. I confirmed this when I use this command:

openssl s_client -connect 149.154.150.134:443

Nginx uses Server Name Indication (SNI) to select the appropriate server block by server_name when these blocks share the same IP address and port.

To test your server using openssl s_client with SNI, you need to provide a server name using the -servername parameter.

For example:

openssl s_client -connect 192.0.2.1:443 -servername 192.0.2.1

In the absence of a -servername parameter, Nginx will direct the request to the default server which will be the first server block in your configuration with the correct listen directive, or the server block explicitly marked with the default_server option.

Also, note that the server_name directive accepts a space separated list of host names - not comma separated. For example:

server_name example.com www.example.com;