Prevent Cloud Run from passing on Authorization header

I have Grafana hosted on Cloud Run. Grafana has anonymous access enabled while Cloud Run needs valid credentials.

Now I try mimic the first example of https://grafana.com/docs/grafana/latest/http_api/dashboard/. Thus I downloaded a service account key to /tmp/creds.json. This file I set as GOOGLE_APPLICATION_CREDENTIALS.
Using these credentials I then send a POST to /api/dashboards/db.
What I see is that I get through Google auth, it starts up Cloud Run instances and I get an error message from Grafana:

{"message":"invalid API key"}

This in turn then lets me believe that Google auth is not consuming the Authorization header but delegating it along.

So my question is how can I prevent Google auth from delegate this header?
Or as a workaround how can I make Grafana ignore the header?

Here the source I use to make the requests:

func TestCall(t *testing.T) {
	r := strings.NewReader(`
	{
	  "dashboard": {
		"id": null,
		"uid": null,
		"title": "Production Overview",
		"tags": [ "templated" ],
		"timezone": "browser",
		"schemaVersion": 16,
		"version": 0,
		"refresh": "25s"
	  },
	  "folderId": 0,
	  "folderUid": "l3KqBxCMz",
	  "message": "Made changes to xyz",
	  "overwrite": false
	}
	`)
	os.Setenv("GOOGLE_APPLICATION_CREDENTIALS", "/tmp/creds.json")
	ctx := context.TODO()
	ts, err := idtoken.NewTokenSource(ctx, "https://grafana-123.run.app")
	if err != nil {
		if !strings.HasPrefix(err.Error(), "idtoken: credential must be service_account") {
			panic(err)
		}
		creds, err := google.FindDefaultCredentials(ctx)
		if err != nil {
			panic(err)
		}
		ts = creds.TokenSource
	}
	tok, err := ts.Token()
	if err != nil {
		panic(err)
	}
	if tok.AccessToken == "" {
		panic("Empty token!")
	}
	fmt.Printf("AT %s\n", tok.AccessToken)
	client := &http.Client{}
	req, err := http.NewRequest("POST", "https://grafana-123.run.app/api/dashboards/db", r)
	if err != nil {
		panic(err)
	}
	req.Header.Set("Authorization", "Bearer "+tok.AccessToken)
	// This does not work at all: req.Header.Set("Proxy-Authorization", "Bearer "+tok.AccessToken)
	req.Header.Set("Content-Type", "application/json")
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
		fmt.Printf("SC %d\n", resp.StatusCode)
		bs, err := io.ReadAll(resp.Body)
		if err != nil {
			panic(err)
		}
		panic(string(bs))
	}
}

The header Proxy-Authorization should work, but I have also been unsuccessful.

To clarify your problem, you are using IAP (OIDC Identity Token) to authorize access to Cloud Run. Your software (Grafana) is processing the HTTP Authorization header and assuming that you are passing it an API Key (which fails). I am not familiar with Grafana internals, but one possible solution is to write middleware that detects an OIDC token and removes that header. I am not aware of a Cloud Run method to remove the Authorization header from HTTP requests that your application receives.

IMHO this is a problem that should be resolved in Grafana's code. If Grafana has anonymous access enabled, then it should not be validating the HTTP Authorization header. Consider opening an issue with Grafana or fixing the code and submitting a patch.