Our Server can't handle more than 20 requests/second

So after 3 months of hard work in developing & switching the company API from PHP to Go I found out that our Go server can't handle more than 20 req/second.

So basically how our API works:

• takes in a request
• validates the request
• fetches the data from the DB using MYSQL
• put's the data in a Map
• send's it back to the Client in a JSON format

So after writing about 30 APIs I decided to take it for a spin and see how it performance under load test.

Test 1: ab -n 1 -c 1 http://localhost:8000/sales/report the results are "Time per request: 72.623 [ms] (mean)" .

Test 2: ab -n 100 -c 100 http://localhost:8000/sales/report the results are "Time per request: 4548.155 [ms] (mean)" (No MYSQL errors).

How did the number suddenly spike from 72.623 to 4548 ms in the second test?. We expect thousands of requests per day so I need to solve this issue before we finally release it . I was surprised when I saw the numbers ; I couldn't believe it. I know GO can do much better.

So basic info about the server and settings:

• Using GO 1.5
• 16GB RAM
• GOMAXPROCS is using all 8 Cores
• db.SetMaxIdleConns(1000)
• db.SetMaxOpenConns(1000) (also made sure we are using pool of
  connections)
• Connecting to MYSQL through unix socket
• System is running under Ubuntu

External libraries that we are using:

• github.com/go-sql-driver/mysql

• github.com/gorilla/mux

• github.com/elgs/gosqljson

Any ideas what might be causing this? . I took a look at this post but didn't work as I mentioned above I never got any MYSQL error. Thank you in advance for any help you can provide.

Your post doesn't have enough information to address why your program is not performing how you expect, but I think this question alone is worth an answer:

> How did the number suddenly spike from 72.623 to 4548 ms in the second test?

In your first test, you did one single request (-n 1). In your second test, you did 100 requests in flight simultaneously (-c 100 -n 100).

You mention that your program communicates to an external database, your program has to wait for that resource to respond. Do you understand how your database performs when you send it 1,000 requests simultaneously? You made no mention of this. Go can certainly handle many hundreds of concurrent requests a second without breaking a sweat, but it depends what you're doing and how you're doing it. If your program can't complete requests as fast as they are coming in, they will pile up, leading to a high latency.

Neither of those tests you told us about are useful to understand how your server performs under "normal" circumstances - which you said would be "thousands of requests per day" (which isn't very specific, but I'll take to mean, "a few a second"). Then it would be much more interesting to look at -c 4 -n 1000, or something that exercises the server over a longer period of time, with a number of concurrent requests which is more like what you expect to get.

I'm not familiar with gosqljson package, but you say your "query by itself is not really complicated" and you're running it against "a well structured DB table," so I'd suggest dropping the gosqljson and binding your query results to structs, then marshalling those structs to json. That should be faster and incur less memory thrashing than using a map[string]interface{} for everything.

But I don't think gosqljson could possibly be that slow, so it's likely not the main culprit.

The way you're doing your second benchmark is not helpful.

> Test 2: ab -n 100 -c 100 http://localhost:8000/sales/report

That's not testing how fast you can handle concurrent requests so much as it's testing how fast you can make connections to MySQL. You're only doing 100 queries and using 100 requests, which means Go is probably spending most of its time making up to 100 connections to MySQL. Go probably doesn't even have time to reuse any of the db connections, considering all the other stuff it's doing to satisfy each query, and then, boom, the test is over. You would need to set the max connections to something like 50 and run 10,000 queries to see how long concurrent requests take once a pool of db connections is already established; right now you're basically testing how long it takes Go to build up a pool of db connections.