Postgres inside docker compose refusing connection from my server

I have a Golang server alongside Postgres instance running inside docker compose. For some reason the Postgres is refusing connection. From all of my previous searches, usually the problem is typo, not exposing the port, having SSL and so on, but I don't have anything like that going on and still having this issue

version: "3.2"
services:
  ingress:
    image: jwilder/nginx-proxy
    ports:
      - "3000:80"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
  auth-service:
    depends_on:
      - rabbitmq
      - auth-db
      - ingress
    build: ./auth
    container_name: "auth-service"
    ports:
      - 3001:3000
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_HOST=auth-db
      - POSTGRES_DB=auth-dev
      - POSTGRES_PORT=5435
      - PORT=3000
      - RABBITMQ_USER=guest
      - RABBITMQ_PASSWORD=guest
      - RABBITMQ_HOST=rabbitmq
      - RABBITMQ_PORT=5672
      - VIRTUAL_HOST=api.twitchy.dev
      - VIRTUAL_PATH=/v1/auth/
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    # networks:
    #   - rabbitmq_net
    #   - default
  rabbitmq:
    image: rabbitmq:3-management-alpine
    container_name: "rabbitmq"
    ports:
      - 5672:5672
      - 15672:15672
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq/
      - rabbitmq_log:/var/log/rabbitmq/
    # networks:
    #   - rabbitmq_net
  auth-db:
    image: postgres:14.1-alpine
    restart: always
    container_name: "auth-db"
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=auth-dev
    ports:
      - "5435:5432"
    volumes:
      - db:/var/lib/postgresql/data
  chat-db:
    image: postgres:14.1-alpine
    restart: always
    container_name: "chat-db"
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_DB=chat-dev
    ports:
      - "5434:5432"
    volumes:
      - db:/var/lib/postgresql/data

# networks:
#   rabbitmq_net:
#     driver: bridge

volumes:
  db:
    driver: local
  rabbitmq_data:
  rabbitmq_log:

This is the error I am getting

auth-service    | Retrying connection to database...
auth-service    | failed to connect to `host=auth-db user=postgres database=auth-dev`: dial error (dial tcp 172.23.0.3:5435: connect: connection refused)

And my Golang code used to connect to the DB (using pgx)

	dbUrl := fmt.Sprintf("postgres://%s:%s@%s:%s/%s?sslmode=disable",
		os.Getenv("POSTGRES_USER"),
		os.Getenv("POSTGRES_PASSWORD"),
		os.Getenv("POSTGRES_HOST"),
		os.Getenv("POSTGRES_PORT"),
		os.Getenv("POSTGRES_DB"))

This is why I am confused

1. The ports match up, I expose 5435 from postgres, and I connect to 5435
2. The host should be correct as I am referencing the auth-db service name and they are on the same default network so that should be fine
3. The password and username match up
4. The POSTGRES_DB also match up, the default database should be auth-dev

POSTGRES_DB
This optional environment variable can be used to define a different name for the default database that is created when the image is first started. If it is not specified, then the value of POSTGRES_USER will be used.

5. I have sslmode disable as well

Is there anything else that can cause the connection to be refused?

Tried changing db to template1 and postgres as they are created by default but both aren't working either

54511e50369c   postgres:14.1-alpine           "docker-entrypoint.s…"   16 minutes ago   Up 16 seconds   0.0.0.0:5435->5432/tcp, :::5435->5432/tcp                                                                                                             auth-db


docker exec -it 54511e50369c psql -U postgres
psql (14.1)
Type "help" for help.

postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
(3 rows)

The database is ready when I am trying to connect (I am retrying 20 times so, and restarting the service if it crashes, so it should be available)

When you map ports in docker-compose, say like "5435:5432", you are mapping port 5435 on the HOST machine to 5432 on the CONTAINER. However, your db url in the auth-service definition is using the name of the service, auth-db, so you are actually hitting the db container directly, not going through the host machine. Because the db container does not expose 5435, you are unable to connect using port 5435.
If you were to try to connect to the database from your host machine for example, you would probably be successful using port 5435 and localhost.