gcloud app deploy cannot find app.yaml if go.mod is in a different directory

I have been unable to deploy my Go app to Google App Engine Flexible for a weeks now, although I don't remember changing anything relevant. The go.mod and go.sum files are in the top directory, while the app.yaml is in ./cmd/app/.

Error message on gcloud app deploy:

Services to deploy:

descriptor:                  [/Users/me/go/src/github.com/me/project/cmd/app/app.yaml]
source:                      [/Users/me/go/src/github.com/me/project/cmd/app]
target project:              [project-us-central1]
target service:              [default]
target version:              [20220118t234216]
target url:                  [https://project-us-central1.uc.r.appspot.com]
target service account:      [App Engine default service account]


Do you want to continue (Y/n)?  

Beginning deployment of service [default]...

ERROR: (gcloud.app.deploy) Required file is not uploaded: [app.yaml]. 
This file should not be added to an ignore list 
(https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore)

(Of course, app.yaml has been never on the ignore list.)

I tried the following:

1. Add !app.yaml to .gcloudignore file
2. Delete .gcloudignore file
3. gcloud config set gcloudignore/enabled false
4. Wait until a new version of gcloud is released (Google Cloud SKD 368.0.0).

Note that this happens both to my existing Go app which was deployed successfully many times over the last few years (the last time in November 2021), and also to a newly created simple web server app under the same directory structure. However, if I create a simple app without Go modules (go.mod, so.sum), gcloud app deploy works as expected.

I just found a workaround while posting this question. Initially, this seems to be working:

gcloud app deploy --appyaml=app.yaml

However, after about 30 minutes the deployment always fails with the following error:

Updating service [default] (this may take several minutes)...failed.                                                                                  
ERROR: (gcloud.app.deploy) Error Response: [13] The system encountered a fatal error

The deployed version seems to be running but it returns HTTP 404 for any request. After stopping and restarting the version in the Console, it's still non-functional: it returns mostly HTTP 502, and sometimes HTTP 503 and "Service error -27". I can see in the logs that /readiness_check calls all fail with 503, failReason:"null"

After a lot of struggling it turned out that the issue was that the ./cmd/app/app.yaml file was not in the same directory as the ./go.mod and ./go.sum files.

My workaround is renaming go.mod temporarily before invoking gcloud app deploy, and then moving it back once the deployment is done. That way everything works fine again.

My workaround is copying go.mod and go.sum to your working directory from where you are running gcloud app deploy command. This works for me.