Go build is failing for private repo with Dockerfile/go.mod files

Trying to build image using Dockerfile, but seeing following error:

[6/7] RUN go mod download && go mod verify:

#10 4.073 go: github.com/private-repo/repo-name@v0.0.0-20210608233213-12dff748001d: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /app/pkg/mod/cache/vcs/40ae0075df7a81e12f09aaa30354204db332938b8767b01daf7fd56ca3ad7956: exit status 128:

#10 4.073 git@github.com: Permission denied (publickey).  
#10 4.073 fatal: Could not read from remote repository.  
#10 4.073 Please make sure you have the correct access rights
#10 4.073 and the repository exists.
------
executor failed running [/bin/sh -c go mod download && go mod verify]: exit code: 1

Here is my Dockerfile:

#Start from base image 1.16.5:
FROM golang:1.16.5

ARG SSH_PRIVATE_KEY

ENV ELASTIC_HOSTS=localhost:9200
ENV LOG_LEVEL=info

#Configure the repo url so we can configure our work directory:
ENV REPO_URL=github.com/private-repo/repo-name

#Setup out $GOPATH
ENV GOPATH=/app

ENV APP_PATH=$GOPATH/src/$REPO_URL

#/app/src/github.com/private-repo/repo-name/src

#Copy the entire source code from the current directory to $WORKPATH
ENV WORKPATH=$APP_PATH/src
COPY src $WORKPATH
WORKDIR $WORKPATH

RUN mkdir -p ~/.ssh && umask 0077 && echo "${SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa \
&& git config --global url."ssh://git@github.com/private-repo".insteadOf 
https://github.com \
&& ssh-keyscan github.com >> ~/.ssh/known_hosts

#prevent the reinstallation of vendors at every change in the source code
COPY go.mod go.sum $WORKDIR
RUN go mod download && go mod verify

RUN go build -x -o image-name .

#Expose port 8081 to the world:
EXPOSE 8081

CMD ["./image-name"]

In my GO env, I do have GO111MODULE=on & GOPRIVATE=github.com/private-repo/*

Also, I'm able to authenticate at my terminal:
ssh -T git@github.com
Hi private-repo! You've successfully authenticated, but GitHub does not provide shell access.

'go get private-repo-name' is successful.

I build through Dockerfile:

docker build --build-arg SSH_PRIVATE_KEY -t image-name .  

which has command:

RUN go build -x -o image-name .

What I tried:

GO111MODULE="on"  
GONOPROXY="github.com/user-id/*"  
GONOSUMDB="github.com/user-id/*"  
GOPRIVATE="github.com/user-id/*"  
GOPROXY="https://proxy.golang.org,direct"

   
    insteadOf = https://github.com/

> Basically, there are multiple repos under github.com/user-name/ and all of them are private which I want to use.

I would rather use a more specific directive:

 git config --global \
 url."ssh://git@github.com/user-name/*".insteadOf https://github.com/user-name/*

That way, the instead of won't apply to all https://github.com URLs, only to the one matching the private repository for which you need to use SSH.
The OP ios-mxe confirms in the comments it is indeed working as expected.