英文:
Golang Concurrency With HTTP Requests
问题
Golang并发与HTTP请求在运行一段时间后出现错误。我不明白是什么原因导致了这个问题。
错误信息：
2022/08/15 15:32:11 API ErrorPost "https://[redacted]/checknum": read tcp [redacted]:63062->[redacted]:443: wsarecv: 远程主机强制关闭了一个现有的连接。
代码：
package main
import (
"bytes"
"crypto/tls"
"fmt"
"io/ioutil"
"bufio"
"net/http"
"strings"
"sync"
"os"
"log"
)
func checknum(wg *sync.WaitGroup, number string) {
defer wg.Done()
headers := map[string]string{
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36",
"Content-Type": "application/json; charset=UTF-8",
}
var data = []byte("\"number\": \""+number+"\"}")
httpRequest("https://[redacted]/checknum", "POST", data, headers, number)
}
func httpRequest(targetUrl string, method string, data []byte, headers map[string]string, number string) {
request, error := http.NewRequest(method, targetUrl, bytes.NewBuffer(data))
for k, v := range headers {
request.Header.Set(k, v)
}
customTransport := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
DisableKeepAlives: true,
}
request.Close = true
client := &http.Client{Transport: customTransport}
response, error := client.Do(request)
if response != nil {
defer response.Body.Close()
}
if error != nil {
log.Fatal("API Error ",error)
}
body, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal("Body Error ",error)
}
if strings.Contains(string(body), `VALID`) {
fmt.Println("[+]:",number)
} else {
fmt.Println("[-]:",number)
}
}
func main() {
var wg sync.WaitGroup
file, err := os.Open(os.Args[1])
if err != nil {
fmt.Println("无法读取文件", err)
return
}
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
num := strings.TrimSpace(scanner.Text())
if num != "" {
wg.Add(1)
go checknum(&wg, num)
}
}
wg.Wait()
}
注意：我的文件包含近10000行代码。
英文:
Golang Concurrency With HTTP Requests Giving Error After Running Some Times. I Am not understanding what is causing this issue. I am
Error:
2022/08/15 15:32:11 API ErrorPost "https://[redacted]/checknum": read tcp [redacted]:63062->[redacted]:443: wsarecv: An existing connection was forcibly closed by the remote host.
Code:
package main
import (
"bytes"
"crypto/tls"
"fmt"
"io/ioutil"
"bufio"
"net/http"
"strings"
"sync"
"os"
"log"
)
func checknum(wg *sync.WaitGroup, number string) {
defer wg.Done()
headers := map[string]string{
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36",
"Content-Type": "application/json; charset=UTF-8",
}
var data = []byte("\"number\": \""+number+"\"}")
httpRequest("https://[redacted]/checknum", "POST", data, headers, number)
}
func httpRequest(targetUrl string, method string, data []byte, headers map[string]string, number string) {
request, error := http.NewRequest(method, targetUrl, bytes.NewBuffer(data))
for k, v := range headers {
request.Header.Set(k, v)
}
customTransport := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
DisableKeepAlives: true,
}
request.Close = true
client := &http.Client{Transport: customTransport}
response, error := client.Do(request)
if response != nil {
defer response.Body.Close()
}
if error != nil {
log.Fatal("API Error ",error)
}
body, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatal("Body Error ",error)
}
if strings.Contains(string(body), `VALID`) {
fmt.Println("[+]:",number)
} else {
fmt.Println("[-]:",number)
}
}
func main() {
var wg sync.WaitGroup
file, err := os.Open(os.Args[1])
if err != nil {
fmt.Println("cannot able to read the file", err)
return
}
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
num := strings.TrimSpace(scanner.Text())
if num != "" {
wg.Add(1)
go checknum(&wg, num)
}
}
wg.Wait()
}
Note: My File Contains Almost 10000 Lines.
答案1
得分: 5
答案就在你的问题中。
wsarecv: 远程主机强制关闭了一个现有的连接。
你的网络服务器正在关闭连接，很可能是因为你给它带来了过多的负载。
如果你有权限访问这个服务器，那么请检查日志。如果你不是这个服务器的所有者，那么你应该像一个更好的网络公民一样，限制对服务器的并发连接数。
你在每次调用时都声明了一个自定义传输方式。最好是只声明一次，并在所有连接中都使用它。
如果你这样做，假设服务器支持，你可能会从http2中受益，连接可以被重用、共享和多路复用。
英文:
The answer is right there in your question.
wsarecv: An existing connection was forcibly closed by the remote host.
Your webserver is closing connections on you, likely because you are overwhelming it.
If you have access to this server, then check the logs. If you do not own this server then you should be acting like a better netizen and limiting the number of concurrent connections to the server.
You are declaring a custom transport for every call. It would be better to declare this once and use it for all connections.
If you do this, then you may benefit from http2, assuming the server supports it, and connections can be reused, shared, and multiplexed.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论