Caching Go Depencencies in Docker

I've read articles in which it is advised to have a dedicated go mod download layer for downloading go dependencies. I understand that the layer is cached and if the dependencies don't change in the next build, the cached layer will be used, saving us time.
I've also noticed that using BuildKit mounting feature, the go build binary results are cached in GO env GOCACHE directory, in order to be used for future builds.

Now the thing is a lot of the time only a few of the go dependencies are changed, therefore the cached go mod download layer is invalidated. And as a result, in the next build, all the dependencies will be downloaded again, even if only a few of them have changed.
My question is if is ok to mount the downloaded packages (dependencies)? And what would be the downsides of doing that? I haven't seen this technique to be used in Go Dockerfile best practices.
Something like

RUN --mount=type=cache,target=/go/pkg/mod go mod download

I think you can just start doing that and see. For me, if I had to use this, I would choose a different mount path, just to isolate the local environment.

RUN --mount=type=cache,target=/Users/me/Library/Caches go mod download

I don't see any problem mounting downloaded packages. In this example, it is used for apt.

However, I think we can also take a step back and consider different approaches to not to be bothered by waiting for Docker build.

We can have scheduled builds (nightly build and etc), I don't see any reason to do local docker build that frequent, especially your dependencies shouldn't change that often. Maybe I am wrong, but I don't think we need to.

We can also further break down go mod download and leverage Docker build cache
E.g.

RUN go mod download package-never-change
RUN go mod download package-changes-frequently

Even

RUN go build ./package1
RUN go build ./package2

It maybe look a little bit tedious, but under certain circumstances it could be useful, for example, when BuildKit is not supported.