解决模拟流的基准函数中的数据竞争问题

huangapple go评论98阅读模式
英文:

Solving data race in benchmark function that simulates stream

问题

我正在尝试编写一个函数,用于对流式传输的CSV数据进行基准测试,并将其发送到HTTP端点。

为了做到这一点,我想生成数据并将其POST出去。

然而,go的数据竞争检测器显示存在数据竞争,并且基准测试完成得比我认为的合理时间要快,所以我猜测HTTP请求没有被正确处理。

我应该如何组织我的测试代码以避免这个问题?

有没有一种方法可以等待HTTP客户端调用被处理完毕?

func BenchmarkStream(b *testing.B) {
    header := "header\n"
    buf := bytes.NewBufferString(header)

    var wg sync.WaitGroup
    wg.Add(1)
    go func() {
        for i := 0; i < b.N; i++ {
            buf.WriteString(fmt.Sprintf("%d\n", i+1))
        }
        wg.Done()
    }()                 // 这一行在数据竞争检测器中被提到

    w := httptest.NewRecorder()
    r, _ := http.NewRequest("POST", "/", buf)
    h := &MyHandler{}
    h.ServeHTTP(w, r)  

    wg.Wait()

    if w.Code != 200 {
        b.Errorf("test failed")
    }
}

编辑:@Grzegorz Żurs的评论让我对我的方法产生了疑问,我使用了io.Pipe进行了重构:

func BenchmarkStream(b *testing.B) {
    pr, pw := io.Pipe()
    go func() {
        pw.Write([]byte("header\n"))

        for i := 0; i < b.N; i++ {
            pw.Write([]byte(fmt.Sprintf("%d\n", i+1)))
        }
    }()

    w := httptest.NewRecorder()
    r, _ := http.NewRequest("POST", "/", pr)
    h := &MyHandler{}
    h.ServeHTTP(w, r)  
    
    if w.Code != 200 {
        b.Errorf("test failed")
    }
}
英文:

I'm trying to write a function that basically benchmarks streamed CSV over to a HTTP endpoint.

To do this I want to generate data and POST that data.

However, gos data race detector says that there is a data race and the benchmark finishes faster than I would believe is reasonable, so I guess the HTTP request is not properly processed.

How should I structure my test code to avoid this?

Is there a way to wait until the HTTP client call has been processed?

func BenchmarkStream(b *testing.B) {
	header := &quot;header\n&quot;
	buf := bytes.NewBufferString(header)

	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		for i := 0; i &lt; b.N; i++ {
			buf.WriteString(fmt.Sprintf(&quot;%d\n&quot;, i+1))
		}
		wg.Done()
	}()                 &lt;-- this line is mentioned in the data race detector

	w := httptest.NewRecorder()
	r, _ := http.NewRequest(&quot;POST&quot;, &quot;/&quot;, buf)
    h := &amp;MyHandler{}
	h.ServeHTTP(w, r)  

	wg.Wait()

	if w.Code != 200 {
		b.Errorf(&quot;test failed&quot;)
	}
}

EDIT: @Grzegorz Żurs comment made me question my approach to begin with, I refactored it with an io.Pipe:

func BenchmarkStream(b *testing.B) {
    pr, pw := io.Pipe()
	go func() {
	    pw.Write([]byte(&quot;header\n&quot;))

		for i := 0; i &lt; b.N; i++ {
			pw.Write([]byte(fmt.Sprintf(&quot;%d\n&quot;, i+1)))
		}
	}()

	w := httptest.NewRecorder()
	r, _ := http.NewRequest(&quot;POST&quot;, &quot;/&quot;, pr)
    h := &amp;MyHandler{}
	h.ServeHTTP(w, r)  
    
	if w.Code != 200 {
		b.Errorf(&quot;test failed&quot;)
	}
}

答案1

得分: 1

你正在两个 goroutine 之间共享 buf

英文:

You are sharing buf between two goroutines.

答案2

得分: 1

如果你只调用处理程序一次,是不可能得到有用的基准结果的。你需要先构建请求体,然后重复调用处理程序。

buf := &bytes.Buffer{}
buf.WriteString("header\n")
buf.WriteString(strings.Repeat("1\n", 1000))
body := buf.Bytes()

b.ResetTimer()

for i := 0; i < b.N; i++ {
    w := httptest.NewRecorder()
    r, err := http.NewRequest("POST", "/", bytes.NewReader(body))
    if err != nil {
        b.Fatal(err)
    }

    h := &MyHandler{}
    h.ServeHTTP(w, r)  
    
    if w.Code != 200 {
        b.Errorf("test failed")
    }
}

希望对你有帮助!

英文:

You're not going to get useful benchmark results if you only invoke the handler once. Build the request body once and then invoke your handler over and over again.

buf := &amp;bytes.Buffer{}
buf.WriteString(&quot;header\n&quot;)
buf.WriteString(strings.Repeat(&quot;1\n&quot;, 1000)
body := buf.Bytes()

b.ResetTimer()

for i := 0; i &lt; b.N; i++ {
    w := httptest.NewRecorder()
    r, err := http.NewRequest(&quot;POST&quot;, &quot;/&quot;, bytes.NewReader(body))
    if err != nil {
        b.Fatal(err)
    }

    h := &amp;MyHandler{}
    h.ServeHTTP(w, r)  

    if w.Code != 200 {
        b.Errorf(&quot;test failed&quot;)
    }
}

huangapple
  • 本文由 发表于 2017年8月25日 21:08:12
  • 转载请务必保留本文链接:https://go.coder-hub.com/45882192.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定