gcloud app deploy fails "cannot import internal package"

I am compiling a GO app that I want to upload and run on the Google Cloud Platform. I am importing the appengine/datastore package and am running into problems with vendoring of the packages. Since I want to provide stable builds I want to have as much of the dependencies vendored in my source tree, but when I vendor appengine/datastore I run in to problems runnning gcloud app deploy:

OperationError: Error Response: [9] Deployment contains files that cannot be compiled: Compile failed:
2017/09/19 01:07:31 go-app-builder: Failed parsing input: package "vendor/google.golang.org/appengine/search" cannot import internal package "google.golang.org/appengine/internal/search"

ERROR: (gcloud.app.deploy) Error Response: [9] Deployment contains files that cannot be compiled: Compile failed:
2017/09/19 01:07:31 go-app-builder: Failed parsing input: package "vendor/google.golang.org/appengine/search" cannot import internal package "google.golang.org/appengine/internal/search"

I can run the dev_appserver.py script just fine, the application runs smoothly locally, and go test succeeds compiling and running all the module tests.

If I try to remove the vendoring of any of the appengine packages and instead use go get to install them outside of the version control, dev_appserver.py no longer runs, complaining about duplicate packages:

rm -rf ../vendor/google.golang.org/appengine
go get google.golang.org/appengine
dev_appserver.py app.yaml
[....]
2017/09/19 10:20:10 go-app-builder: Failed parsing input: package "golang.org/x/net/context" is imported from multiple locations: "/home/peter/src/myproject/go/src/myproject/vendor/golang.org/x/net/context" and "/home/peter/src/myproject/go/src/golang.org/x/net/context"

while gcloud app deploy instead complains about not finding the packages at all:

[...]
File upload done.
Updating service [default]...failed.                                                             
ERROR: (gcloud.app.deploy) Error Response: [9] Deployment contains files that cannot be compiled: Compile failed:
Compile failed:
2017/09/19 01:22:13 go-app-builder: build timing: 7×compile (1.749s total), 0×link (0s total)
2017/09/19 01:22:13 go-app-builder: failed running compile: exit status 2
myproject/vendor/golang.org/x/text/unicode/norm/normalize.go:15: can't find import: "golang.org/x/text/transform"
$ find .. -name transform
../vendor/golang.org/x/text/transform

EDIT: WORKAROUND: I have found that I can make it compile with gcloud by symlinking the vendored directories (github.com and golang.org) into the application directory (ln -s ../vendor/* .), and downloading the appengine package manually (go get google.golang.org/appengine). However, I need to delete the symlinks to be able to run dev_appserver.py, so this is not nearly optimal.

The reason this happens is because of how App Engine builds Go apps.

Because of the App Engine standard environment security sandbox and container model, they prohibit Go code from importing any packages which could potentially mess with their systems--like unsafe, for instance. They also block direct imports of the internal appengine packages (google.golang.org/appengine/internal/whatever) for the same reason.

For this project, you've locally vendored your dependencies. For the app engine app builder, though, this looks no different then if you had just made another sub-package in your app and put the dependencies in there. It's functionally the same as if you had just copied and pasted appengine/search straight into your project.

Because appengine/search imports its corresponding internal package, appengine/internal/search, and the Go app builder doesn't really differentiate between what's in your application code and what's in your vendor directory, it fails the build. To App Engine, it looks like you imported appengine/internal/search, which is prohibited by the security model. It doesn't know how appengine/internal/search is being used, because it doesn't control what's in vendor, so to keep the sandbox secure it doesn't let you import it.

The solution to this problem is, basically, to not vendor your dependencies.

If you remove all the appengine/whatever packages from vendor, then the App Engine builder will look for them on Google's build servers instead of in your project. The App Builder trusts its own local copies of the appengine libraries not to misbehave, so they're permitted to import appengine/whatever/internal packages.

(As an aside, if you haven't already, make sure gcloud and all related components are up to date. This can sometimes resolve gnarly dependency issues)