How could I manage the App Engine Go runtime context to avoid App Engine lock-in?

I'm writing a Go application to run on App Engine's Go runtime.

I notice that pretty much any operation which uses an App Engine service (such as Datastore, Mail, or even Capabilities) requires that you pass it an instance of appengine.Context which must be retrieved using the function appengine.NewContext(req *http.Request) Context.

While I am writing this app for App Engine, I want to be able to move it to some other platform (possibly one which doesn't support any of the App Engine API) easily and quickly if I should so choose.

So, I'm abstracting away the actual interaction with App Engine services and API's by writing little wrappers around any App-Engine-specific interaction (including request handling functions). With this approach, if I ever do wish to move to a different platform, I'll just rewrite those specific modules which tie my application to App Engine. Easy and straightforward.

The only problem is that appengine.Context object. I can't pass it down from my request handlers through my layers of logic to the modules which handle these API's without tying pretty much all of my code to App Engine. I could pass the http.Request object from which the appengine.Context object can be derived, but that would require coupling things that probably shouldn't be coupled. (I think it's best practice for none of my application to even know it's a web application except those portions specifically dedicated to handling HTTP requests.)

The first solution that sprang to mind was to just create a persistent variable in some module. Something like this:

package context

import (
    "appengine"
)

var Context appengine.Context

Then, in my request handlers, I can set that variable with context.Context = appengine.NewContext(r) and in the modules that directly use App Engine services, I can fetch the context by accesing context.Context. None of the intervening code would need to know of the appengine.Context object's existence. The only problem is that "multiple requests may be handled concurrently by a given instance", which can lead to race conditions and unexpected behavior with this plan. (One request sets it, another sets it, the first one accesses it and gets the wrong appengine.Context object.)

I could in theory store the appengine.Context to datastore, but then I'd have to pass some request-specific identifier down the logic layers to the service-specific modules identifying which appengine.Context object in datastore is the one for the current request, which would again couple things I don't think should be coupled. (And, it would increase my application's datastore usage.)

I could also pass the appengine.Context object down the whole logic chain with the type interface{} the whole way and have any module which doesn't need the appengine.Context object ignore it. That would avoid tying most of my application to anything specific. That also seems highly messy, however.

So, I'm at a bit of a loss how to cleanly ensure the App-Engine-specific modules which need the appengine.Context object can get it. Hopefully you folks can give me a solution I have yet to think of myself.

Thanks in advance!

This is tricky because your self-imposed scoping rule (which is a sensible one) means not passing a Context instance around, and there is nothing similar to Java's ThreadLocal to achieve the same ends by sneaky means. That's actually a Good Thing, really.

Context combines logging support (easy) with a Call to appengine services (not easy). There are I think ten appengine functions that need a Context. I can't see any clean solution other than wrapping all of these behind your own facade.

There is one thing that can help you - you can include a configuration file with your app that indicates whether it's in GAE or otherwise, using a flag of some sort. Your global boolean need only store this flag (not a shared context). Your facade functions can then consult this flag when deciding whether to use NewContext(r) to obtain the Context to access GAE services, or use a lookalike structure to access your own substitute services.

Edit: As a final remark, when you solve this may I invite you to share how you did it, possibly even with an open-source project? Cheeky of me to ask, but if you don't ask...

I (hopefully) solved this issue by wrapping my request handlers (in this example one called "realHandler") like this:

http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    ds := NewDataStore(r)
    realHandler(w, r, ds)
})

NewDataStore creates a DataStore, which is a simple wrapper that abstracts the GAE datastore. It has an unexposed field where it stores the context:

type DataStore struct {
    c appengine.Context
}

func NewDataStore(req *http.Request) *DataStore {
    return &DataStore{appengine.NewContext(req)}
}

Within the request handler, I can just access the abstracted datastore when I need it without worrying about the GAE context, which has already been set:

func realHandler(w http.ResponseWriter, req *http.Request, db *DataStore) {
    var s SomeStruct{}
    key, err := db.Add("Structs", &s)
    ...
}

Particularly in case of Datastore you should be able to reuse the same appengine.Context among different requsts. I haven't tried to do it myself but that's the way an alternative API for Datastore called goon works:

> Goon differs from the datastore package in various ways: it remembers the appengine Context, which need only be specified once at creation time

The fact that storage should depend on HTTP request sounds ridiculous by the way. I don't think Datastore depends on a particular request in the usual sense. Most probably, it's needed to identify a particular Google App Engine application which obviously stays the same from request to request. My speculations are based on quick skim over the source code of google.golang.org/appengine.

You may do similar observations in regard of the other Google App Engine APIs. Of cause all the details could be implementation specific and I'd performed more deep research before actually using those observations in a real application.

It's worth to note that Go team introduced a golang.org/x/net/context package.

Later on the context was made available in Managed VMs, the repo is here. The documentation states:
> This repository supports the Go runtime on App Engine, including both classic App Engine and Managed VMs. It provides APIs for interacting with App Engine services. Its canonical import path is google.golang.org/appengine.

What it means is that you could easily write another packages out of dev environment depending on appengine.

In particular it becomes very easy to wrap around packages like appengine/log (trivial log wrapper example).

But even more important this allows one to create handlers in a form:

There's an article about a context package on Go blog here. I wrote about using context here. If you decide to use the handler with context passing around it's good to create context for all requrests in one place. You can do it by using non standard requrest router like github.com/orian/wctx.