Getting error related to permission trying to pull docker image from Google Artifact Registry via Container-Optimized OS VM

I am trying to use Cloud Build using Google cloud to pull and run an image on Container-Optimized OS VM compute engine
In my cloudbuild.yaml I have

steps:

  - name: 'gcr.io/cloud-builders/gcloud'
    args:
      - compute
      - ssh
      - --zone
      - "us-central1-a"
      - $_SSH_STRING
      - --project
      - $PROJECT_ID
      - --
      - docker pull us-central1-docker.pkg.dev/$PROJECT_ID/us/$_IMAGE_NAME:latest &&
      - docker run -d -p 1338:1338   us-central1-docker.pkg.dev/$PROJECT_ID/us/$_IMAGE_NAME:latest

I run gcloud builds submit --config cloudbuild.yaml .
getting this error

Error response from daemon: Head "https://us-central1-docker.pkg.dev/v2/myProjectId/us/imageName/manifests/latest": denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/myProjectId/locations/us-central1/repositories/us" (or it may not exist)

I am sure that the PROJECTNUMBER-compute@developer.gserviceaccount.com has Artifact Registry Reader and I even tried to give it Artifact Registry Administrator role. More over I made a custom role where it has artifactregistry.repositories.downloadArtifacts added

Still can't make it to work and I get the same error
I also SSH to the VM and tried to run it manually and still the same error about permission. I don't know what more permission I can grant

PS: I am sure the image exists

EDIT:

The error occurs within the VM after Cloud Build establishes an SSH connection to it. This error is not caused by Cloud Build itself, but rather by the permissions of the VM's compute engine "Which as I siad above, I already made sure that the service account has the right roles and permissions". Furthermore, this error can occur not only when using the Container-Optimized OS but also when running an Ubuntu OS. The only difference is that in Ubuntu OS, you have the option to install Google Cloud SDK and run the 'gcloud credentials helper' like this gcloud auth configure-docker $_ARTIFACT_ZONE-docker.pkg.dev.

Posting this as a community wiki for everyone's visibility.

Container-optimized OS loads one specified image as part of the VM configuration. If running images via docker is what you want, then you should use Compute Engine and not Container-optimized OS. The two services appear similar but have very different security models enforced.