Why are images from the GCP Artifact Registry not being pulled by my deployments?

I have two projects in GCP. Let's call them project A and project B. Project A will contain the cluster for the deployment, while project B will contain the Artifact Registry (please note that this is not the same as the Container Registry).

The deployment is pulling a docker image from the Artifact Registry. Let's say the docker image is a stock standard Ubuntu image that our deployment is trying to deploy, and let's assume that the image is already in the Artifact Registry with no issues.

The deployment of the ubuntu image to the cluster is done via a GitLab CI/CD pipeline. It uses a service account that has the following roles:

• Artifact Registry Reader
• Kubernetes Engine Developer
• Viewer

Additionally we also note that the cluster features two node pools. One is a custom node pool and the other is the default. They both have the following access scopes:

• https://www.googleapis.com/auth/cloud-platform
• https://www.googleapis.com/auth/devstorage.read_only
• https://www.googleapis.com/auth/logging.write
• https://www.googleapis.com/auth/monitoring.write
• https://www.googleapis.com/auth/service.management.readonly
• https://www.googleapis.com/auth/servicecontrol

The cloud-platform scope is enabled to ensure that the node pools can pull from the Artifact Registry which is in a different project.

It is also important to note that both node pools use the default service account which has the roles:

• Artifact Registry Reader
• Editor

Upon deployment, the GitLab pipeline completes with no issues. However, the deployment workload fails in the cluster. The following events occur:

1. Pulling image "europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest"
2. Failed to pull image "europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest": rpc error: code = Unknown desc = failed to pull and unpack image "europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest": failed to resolve reference "europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden
3. Error: ErrImagePull
4. Error: ImagePullBackOff
5. Back-off pulling image "europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest"

The deployment YAML file is as follows:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu-build-deploy-demo
  labels:
    app: ubuntu
  namespace: customnamespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ubuntu
  template:
    metadata:
      labels:
        app: ubuntu
    spec:
      containers:
        - name: ubuntu
          image: europe-west1-docker.pkg.dev/B/docker-repo/ubuntu:latest
          command: ["sleep", "123456"]

Why is the image not pulled from correctly? Why am I getting an auth issue despite the correct service account roles and access scopes? How can I resolve this issue?

I have double checked that the image name, tag, and path are correct many times. I have also double checked that the service accounts have the correct roles (specified earlier). I have also ensured that the node pool access scopes are correct and do indeed have the correct access scopes (specified earlier).

I am at a loss of how to resolve this issue. Any help would be greatly appreciated.

Putting in answer form. As @avinashjha said: the default service account from project A needs to have the role Artifact Registry Reader in project B. This allows for the default service account to oauth and pull the docker image from the registry.