Go mod tidy from internal repo via ssh failing

I cannot go mod tidy when using an internal GitHub repo. SSH auth seems to work in isolation (see log below) but the go mod tidy hits the internal repo and throws

git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.

From my Github action

steps:
            # Checks-out the repository under $GITHUB_WORKSPACE, so the job can access it
            - uses: actions/checkout@v3

            # Set up Go
            - name: Set up Go
              uses: actions/setup-go@v4
              with:
                  go-version: 1.20.2
                  cache: false

            - name: Add private key to SSH agent
              env:
                PRIVATE_KEY: ${{ secrets.SHARED_DEPLOY_PRIVATE_KEY }}
              run: |
                echo "${{ env.PRIVATE_KEY }}" > key.pem
                chmod 600 key.pem
                eval "$(ssh-agent -s)"
                ssh-add key.pem
                ssh-add -l -E sha256
                ssh -T git@github.com 2>&1 || true    ### tests ssh auth

            # Install dependencies
            - name: Install dependencies
              run: |
                  git config --global url."git@github.com:".insteadOf "https://github.com/"
                  go clean -modcache  
                  go env -w GOPRIVATE=github.com/my-org/*
                  go env -w GONOPROXY=github.com/my-org/*
                  go env
                  go mod tidy

Log for Add private key

Agent pid 1766
Identity added: key.pem (_REDACTED_)
3072 SHA256:_REDACTED_ _REDACTED_ (RSA)
Hi my-org/observability-go! You've successfully authenticated, but GitHub does not provide shell access.

Log for Install dependencies

go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/mattn/go-isatty v0.0.17
go: downloading golang.org/x/sync v0.1.0
github.com/my-org/clan-service/cmd/clanservice imports
	github.com/my-org/observability-go/logging: github.com/my-org/observability-go@v0.0.0-20230623103942-2be438a81907: invalid version: git ls-remote -q origin in /home/runner/go/pkg/mod/cache/vcs/d0c7f50097d6054d27fc7949420737cdb6036d1246584bb05f13c6fe75577be2: exit status 128:
	git@github.com: Permission denied (publickey).
	fatal: Could not read from remote repository.
	Please make sure you have the correct access rights
	and the repository exists.

Output from go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/runner/.cache/go-build"
GOENV="/home/runner/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/runner/go/pkg/mod"
GONOPROXY="github.com/my-org/*"
GONOSUMDB="github.com/my-org/*"
GOOS="linux"
GOPATH="/home/runner/go"
GOPRIVATE="github.com/my-org/*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/hostedtoolcache/go/1.20.2/x64"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/hostedtoolcache/go/1.20.2/x64/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/runner/work/clan-service/clan-service/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4021260014=/tmp/go-build -gno-record-gcc-switches"

What have I missed? How do I make this work?

Each step is run in a separate process, so actions which modify the environment for the scope of the current process will not "survive" to the next step.

For example:
you would need to re-run eval "$(ssh-agent -s)" in your "Install dependencies" step, so that the environment is set up correctly to talk to your ssh agent.