multi thread requesting in go and not getting high RPS

I'm trying to write a multi thread client to test my server. when I use 2 goroutines everything is fine and I get 50k RPS and my CPU load is normal but when I create more than 2 the RPS drop down to 3K however my CPU load exceed. although when i run my client code multi times (for example run same code on 3 consol at same time) i get more RPS like 80k RPS.

here is my client side code

package main
import (
"fmt"
"net/http"
"os"
"sync"
"time"
)
func main() {
requestURL := fmt.Sprintf("http://localhost:%d/home", 3333)
var wg sync.WaitGroup
wg.Add(4)
req, err := http.NewRequest(http.MethodGet, requestURL, nil)
if err != nil {
fmt.Printf("client: could not create request: %s\n", err)
os.Exit(1)
}
for i := 0; i < 4; i++ {
go func() {
defer wg.Done()
client := http.Client{
Timeout: 30 * time.Second,
}
for {
client.Do(req)
}
}()
}
wg.Wait()
}

and here is my server side code

package main
import (
"errors"
"fmt"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"log"
"net/http"
"os"
"sync"
)
// log handling
func openLogFile(path string) (*os.File, error) {
logFile, err := os.OpenFile(path, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0644)
if err != nil {
return nil, err
}
return logFile, nil
}
// variables of counter in metric
var okStatusCounter = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "ok_request_count",
Help: "Number of 200",
},
)
func listener(serverLog *log.Logger) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
//metric 
okStatusCounter.Inc()
w.WriteHeader(http.StatusOK)
}
}
func main() {
//metric
prometheus.MustRegister(okStatusCounter)
//log handling
fileSimpleServerLog, err := openLogFile("simpleServer/simpleServerLog.log")
if err != nil {
log.Fatal(err)
}
serverLog := log.New(fileSimpleServerLog, "[simple server]", log.LstdFlags|log.Lshortfile|log.Lmicroseconds)
var wg sync.WaitGroup
wg.Add(1)
//server:
go func() {
defer wg.Done()
mux := http.NewServeMux()
mux.HandleFunc("/home", listener(serverLog))
mux.Handle("/metrics", promhttp.Handler())
server := http.Server{
Addr:    fmt.Sprintf(":%d", 3333),
Handler: mux,
}
if err := server.ListenAndServe(); err != nil {
if !errors.Is(err, http.ErrServerClosed) {
serverLog.Printf("error running http server: %s\n", err)
}
}
}()
wg.Wait()
}

at first I thought go may use one port for all of client connection but when I checked it out with netstat it was using multiple ports. I tried to search but I couldn't find any proper answer

I tried sync.Mutex :

var mu sync.Mutex
...
for i := 0; i < 1000; i++ {
go func() {
defer wg.Done()
client := http.Client{
//Timeout: 30 * time.Second,
}
for {
mu.Lock()
_, err := client.Do(req)
if err != nil {
clientLog.Printf("client: error making http request: %s\n", err)
os.Exit(1)
}
mu.Unlock()
}
}()
}
wg.Wait()
...

with change above I get 13k RPS and my CPU load is normal but that's not near enough at all

Since you send requests to only one host, the default values of HTTP Transport are not suitable for you. It is better to set the parameters manually in your case:

t := http.DefaultTransport.(*http.Transport).Clone()
t.MaxIdleConns = 100
t.MaxConnsPerHost = 100
t.MaxIdleConnsPerHost = 100
httpClient = &http.Client{
Timeout:   10 * time.Second,
Transport: t,
}

For more information, you can read here.