read from secondary with mgo.Monotonic

I am trying to configure reading from primary and two secondary nodes of mongo replica set to provide better load balancing. Each of 3 nodes are on different machines with IP addresses: ip1, ip2, ip3.

My GoLang site, which is the martini web server with two urls /insert and /get:

package main
import (
"github.com/go-martini/martini"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
"net/http"
)
const (
dialStr        = "ip1:port1,ip2:port2,ip3:port3"
dbName         = "test"
collectionName = "test"
elementsCount  = 1000
)
var mainSessionForSave *mgo.Session
func ConnectToMongo() {
var err error
mainSessionForSave, err = mgo.Dial(dialStr)
mainSessionForSave.SetMode(mgo.Monotonic, true)
if err != nil {
panic(err)
}
}
func GetMgoSessionPerRequest() *mgo.Session {
var sessionPerRequest *mgo.Session
sessionPerRequest = mainSessionForSave.Copy()
return sessionPerRequest
}
func main() {
ConnectToMongo()
prepareMartini().Run()
}
type Element struct {
I int `bson:"I"`
}
func prepareMartini() *martini.ClassicMartini {
m := martini.Classic()
sessionPerRequest := GetMgoSessionPerRequest()
m.Get("/insert", func(w http.ResponseWriter, r *http.Request) {
for i := 0; i < elementsCount; i++ {
e := Element{I: i}
err := collection(sessionPerRequest).Insert(&e)
if err != nil {
panic(err)
}
}
w.Write([]byte("data inserted successfully"))
})
m.Get("/get", func(w http.ResponseWriter, r *http.Request) {
var element Element
const findI = 500
err := collection(sessionPerRequest).Find(bson.M{"I": findI}).One(&element)
if err != nil {
panic(err)
}
w.Write([]byte("get data successfully"))
})
return m
}
func collection(s *mgo.Session) *mgo.Collection {
return s.DB(dbName).C(collectionName)
}

I run this GoLang site with the command go run site.go and to prepare my experiment requested http://localhost:3000/insert - after about a minute my test data was inserted.

Then I started to test reading from secondary and primary nodes
in attacker.go:

package main
import (
"fmt"
"time"
vegeta "github.com/tsenart/vegeta/lib"
)
func main() {
rate := uint64(4000) // per second
duration := 4 * time.Second
targeter := vegeta.NewStaticTargeter(&vegeta.Target{
Method: "GET",
URL:    "http://localhost:3000/get",
})
attacker := vegeta.NewAttacker()
var results vegeta.Results
for res := range attacker.Attack(targeter, rate, duration) {
results = append(results, res)
}
metrics := vegeta.NewMetrics(results)
fmt.Printf("99th percentile: %s\n", metrics.Latencies.P99)
}

Running it go run attacker.go I just requested URL http://localhost:3000/get 4000 times per second. While attacker was working I opened all my 3 servers and run htop command to watch resources consumption. The PRIMARY node shows that it is under high load with CPU about 80%. The SECONDARIES were calm.

Why?

As I used mgo.Monotonic ...

mainSessionForSave.SetMode(mgo.Monotonic, true)

... I expected to read from all nodes: ip1, ip2, ip3 and I expected to watch all the nodes under equal load and with equal CPU consumption. But it is not so. What did I configure wrong? In fact mgo.Monotonic is not working in my case and I read only from the PRIMARY node.

The sessionPerRequest is only created once: prepareMartini is called at server startup, and sessionPerRequest is set then. The closures passed to m.Get() access that variable. Then, after the first write (during your test setup), mgo will only access the primary:

> Monotonic consistency will start reading from a slave if possible, so that the load is better distributed, and once the first write happens the connection is switched to the master.

(If mgo just continued reading from the secondary after writing to the primary, a read wouldn't necessarily reflect a write you just made, which could be a pain. And switching to the primary should only get you newer data than you were getting from the secondary, never older, which preserves monotonicity. That's how it should ideally work, anyway; see the "open issues" link below for more.)

The solution is to push creating the session down into your handlers, e.g., remove sessionPerRequest and put something explicit atop each handler, like

coll := mainSessionForSave.Copy().DB(dbName).Collection(collName)

All consistency promises should be read in light of open issues with MongoDB consistency: right now, during network partitions, reads can see old data and writes that will later be rolled back, even when mgo is trying to read from the primary. (Compare-and-set doesn't have this issue, but of course that's a larger slower operation.) It's also worth perusing that post just for the discussion of the consistency levels and the descriptions of how different database behaviors might manifest for an app's end-users.

You forget to close connections after use it:

defer mainSessionForSave.Close()

Probably, it may be a reason.

P.S. make sure that all your nodes is available