Upserting to sharded MongoDB using mgo Error "full shard key must be in update object for collection:..."

Using the Labix mgo API for MongoDB, I am trying to perform an increment operation on a sharded collection. I am able to do this just fine on an unsharded collection using the usual mgo.Change struct, but when I try to do this on a sharded collection, I get the error: full shard key must be in update object for collection: db_name.collection_name

The original code that works on an un-sharded collection looks like this:

            change := mgo.Change{
                ReturnNew: true,
                Upsert: true,
                Update: bson.M{
                    "$setOnInsert": bson.M{
                        "ci": r.Ci,
                        "dt": r.Dt,
                        "zi": r.Zi,
                    },
                    "$inc": &data,
                },
            }

            _, err := collection.Upsert(bson.M{"_id": id, "ci": r.Ci, "dt": r.Dt, "zi": r.Zi}, change); if err != nil {
                log.Println("FAILURE", err)
            }

However, when I switch to a sharded collection, sharded on the key {ci: 1, dt: 1, zi: 1} I get the error mentioned above.

In an attempt to debug, I've tried to figure out what is going on behind the scenes with mgo, and tried to do an insert directly into the mongo terminal.

db.collection.update({ "_id" : "98364_2013-12-11", "ci" : "16326", "dt" : "2013-12-11", "zi" : "98364"}, {$setOnInsert: { "ci" : "16326", "dt" : "2013-12-11", "zi" : "98364"} , $inc: {test :1}}, { upsert: true });

However, this got me a separate error: Can't modify shard key's value. field: ci: "16326" collection: db.collection This is something I think I'll have to figure out once I figure out my inital error, but it seems strange to me that its throwing this error with the $setOnInsert command since its not supposed to modify the value, just set it on the initial insert. All errors go away when I cut out the $setOnInsert portion of the command, but I need a way to make sure these values get set because they will be important in the queries I write to get the data back out.

Back to my main problem: I did find that when I rearranged the order of the update and upsert documents when interacting with the MongoDB terminal, I got the error I'm getting when I go through mgo, so I tried to very strictly control the order of the documents being passed in the mgo.Change struct by switching to bson.D:

            change := bson.D{
                {
                     "Update",
                     bson.D{
                         {"$setOnInsert", bson.D{
                                {"_id", id},
                                {"ci", r.Ci},
                                {"dt", r.Dt},
                                {"zi", r.Zi},
                                },
                         },
                         {"$inc", &data},
                     },
                },
                {
                    "Upsert",
                    true,
                },
            }
            log.Println(change)
            err := collection.Update(bson.D{{"_id", id},{ "ci", r.Ci},{ "dt", r.Dt}, {"zi", r.Zi}},change); if err != nil {
                log.Println("FAILURE", err)
            }

At this point, printing the change object yields: [{Update [{$setOnInsert [{_id 11635_2013-12-11} {ci 3599} {dt 2013-12-11} {zi 11635}]} {$inc 0xc21dd9d8d0}]} {Upsert true}] which I believe to be exactly what I should be passing in as the change object in exactly the right order according to Mongo's documentation, but I still get the same full shard key must be in update object for collection: db.collection error.

I realize that using collection.Find({_id: ... }).Apply(change, ...) is a possible alternative and it works properly when I've used it, but in my testing on unsharded collections, I've seen way higher performance (as in ~20x faster) using the Upsert (or Update) functions and speed is absolutely a priority since I'm dealing with tens of thousands of events per second.

I'm getting to the point where I feel like I've tried every think I can think of and would appreciate a fresh set of eyes trying to help me figure out what is happening, so any help would be appreciated.

The mgo.Change type is specific to the Query.Apply method, which runs the MongoDB findAndModify command and does any of the supported modifications at once. The Upsert method, on the other hand, takes a modification document that will be directly provided to mgo/bson for marshalling. These modification documents have the same format whether you provide them via Query.Apply (in the Update field of mgo.Change), or via the Collection.Upsert or Collection.Update methods.

So, the observed error is being caused because it's attempting to use mgo.Change as a plain struct for inserting (in other words, a document with keys "returnnew", etc), which is definitely not what you want. The shell command you provided, for example, is equivalent to the straightforward translation with mgo:

type M map[string]interface{}
err := collection.Upsert(
    M{
        "_id": "98364_2013-12-11",
        "ci":  "16326",
        "dt":  "2013-12-11",
        "zi":  "98364",
    },
    M{
        "$setOnInsert": M{"ci": "16326", "dt": "2013-12-11", "zi": "98364"},
        "$inc":         M{"test": 1},
    },
)

This is still broken, though, but for a different reason. As the server mentioned in the error message, this is attempting to set the shard key a second time. Note that an upsert operation will make use of the fields provided in the query document, and merge both the query document and the modification document to create the final document for insertion. This means that the shard key fields in the $setOnInsert document are redundant with the shard key fields in the query document.

I'll improve the documentation on that area to reduce the chances of people getting confused by the use of mgo.Change. Sorry for the trouble.