Golang程序在进行Get请求后返回HTTP状态码503或待处理。

huangapple go评论105阅读模式
英文:

Golang program after making Get request returns HTTP Status code 503 or pending

问题

我试图通过发送GET请求从Kibana服务器获取CSV数据。
然后服务器会发送带有数据的响应,我将该数据写入CSV文件并保存在某个位置。
每次CSV文件中只有一行"pending"。
Kibana日志显示状态码503。
但是,如果我在浏览器中输入URL,我可以获取带有正确数据的CSV文件。
可能我需要等待更长时间以获取来自Kibana的响应。
从我的角度来看,问题是这样的:服务器可以发送带有一行"pending"的响应,这意味着它需要更多时间来准备正确的响应。
我尝试增加客户端的超时时间,但也没有起作用。

使用Go协程和通道/等待组的思路是:它们强制GET请求等待获取正确的数据,而不是"pending"和503状态码。

以下是代码示例:

package main

import (
	"encoding/json"
	"errors"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"sync"
	"time"
)

type KibanaReportResponse struct {
	Path string `json:"path"`
}

var urlKibanaBase = "http://localhost:5601"
var urlKibanaPost = urlKibanaBase + "/api/reporting/generate/csv_searchsource?"
var urlParameters = "jobParams=(browserTimezone:Europe/Berlin,columns:!(),objectType:search,searchSource:(fields:!((field:'*',include_unmapped:!t)),index:ec074c00-1f62-11ec-8056-8d208a1f6e77,parent:(filter:!(),index:ec074c00-1f62-11ec-8056-8d208a1f6e77,query:(language:kuery,query:'')),sort:!((_score:desc)),trackTotalHits:!t,version:!t),title:'Discover search [2021-09-27T09:19:44.977+02:00]')"

var urlWithParam = urlKibanaPost + urlParameters

func main() {
	var wg sync.WaitGroup
	wg.Add(1)
	pathCsvFile := getCsvPathFromKibana(urlWithParam)
	go getCsvFile(urlKibanaBase, pathCsvFile, &wg)
	defer wg.Wait()
}

func getCsvPathFromKibana(urlKib string) string {
	resKibana := KibanaReportResponse{}
	client := &http.Client{
		Timeout: 10 * time.Second,
	}
	if req, err := http.NewRequest("POST", urlKib, nil); err != nil {
		log.Println("给定的方法、URL和可选的正文错误", err)
	} else {
		req.Header.Add("kbn-xsrf", "true")
		if res, err := client.Do(req); err != nil {
			log.Println("可能是由于客户端策略或HTTP连接的问题", err)
		} else {
			if err := json.NewDecoder(res.Body).Decode(&resKibana); err != nil {
				log.Println("JSON解码问题\n", err)
			} else {
				return resKibana.Path
			}
		}
	}

	return resKibana.Path
}

func getCsvFile(urlKibanaBase string, pathCsvFile string, wg *sync.WaitGroup) error {
	defer wg.Done()

	res, err := http.Get(urlKibanaBase + pathCsvFile)
	if err != nil {
		return err
	}
	defer res.Body.Close()
	switch res.StatusCode {
	case 200:
		dataBody, err := ioutil.ReadAll(res.Body)
		if err != nil {
			return err
		}
		err = ioutil.WriteFile("data.csv", dataBody, 0666)
		if err != nil {
			return err
		}
		return nil
	case 503:
		fmt.Println("可能是503/pending")
		return errors.New("可能是503/pending")
	}

	return nil
}

curl请求示例:

curl -v localhost:5601/api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u

截图显示我可以通过在浏览器中输入URL路径来获取数据。

更新:
我尝试递归调用我的函数,但这是一个不好的解决方案,即使它现在可以工作。

case 503:
	getCsvFile(urlKibanaBase, pathCsvFile)

希望对你有帮助!

英文:

I try to get csv data from kibana server by making get request.
After this server send me response with data which i write the csv file and save somewhere.
<br/>Every time csv file with single line "pending" inside.
<br/>Kibana logs shows Status code 503
<br/>But if i put url in browser i can get csv file with correct data.
<br/>Probably i need to wait more for response from kibana.
<br/>From my perspective the problem is next: server can send me the response with single line "pending" which means that it needs more time for preparing right response.
<br/>I tried to increase client time but it doesn't work as well

client := http.Client{
Timeout: 10 * time.Second,
}

The Idea with go routines and channels/wait group was next: they force Get request to wait for getting right data instead of "pending" and 503 status code


import (
&quot;encoding/json&quot;
&quot;errors&quot;
&quot;fmt&quot;
&quot;io/ioutil&quot;
&quot;log&quot;
&quot;net/http&quot;
&quot;sync&quot;
)
type KibanaReportResponse struct {
Path string `json:&quot;path&quot;`
}
var urlKibanaBase = &quot;http://localhost:5601&quot;
var urlKibanaPost = urlKibanaBase + &quot;/api/reporting/generate/csv_searchsource?&quot;
var urlParameters = &quot;jobParams=%28browserTimezone%3AEurope%2FBerlin%2Ccolumns%3A%21%28%29%2CobjectType%3Asearch%2CsearchSource%3A%28fields%3A%21%28%28field%3A%27%2A%27%2Cinclude_unmapped%3Atrue%29%29%2Cindex%3Aec074c00-1f62-11ec-8056-8d208a1f6e77%2Cparent%3A%28filter%3A%21%28%29%2Cindex%3Aec074c00-1f62-11ec-8056-8d208a1f6e77%2Cquery%3A%28language%3Akuery%2Cquery%3A%27%27%29%29%2Csort%3A%21%28%28_score%3Adesc%29%29%2CtrackTotalHits%3A%21t%2Cversion%3A%21t%29%2Ctitle%3A%27Discover%20search%20%5B2021-09-27T09%3A19%3A44.977%2B02%3A00%5D%27%29&quot;
var urlWithParam = urlKibanaPost + urlParameters
func main() {
var wg sync.WaitGroup
wg.Add(1)
pathCsvFile := getCsvPathFromKibana(urlWithParam)
go getCsvFile(urlKibanaBase, pathCsvFile, &amp;wg)
defer wg.Wait()
}
func getCsvPathFromKibana(urlKib string) string {
resKibana := KibanaReportResponse{}
client := &amp;http.Client{}
if req, err := http.NewRequest(&quot;POST&quot;, urlKib, nil); err != nil {
log.Println(&quot;Given a method, URL, andoptional body are wrong&quot;, err)
} else {
req.Header.Add(&quot;kbn-xsrf&quot;, &quot;true&quot;)
if res, err := client.Do(req); err != nil {
log.Println(&quot;Probably problems depends on client policy or HTTP connection&quot;, err)
} else {
if err := json.NewDecoder(res.Body).Decode(&amp;resKibana); err != nil {
log.Println(&quot;Problem by Json decoding \n&quot;, err)
} else {
return resKibana.Path
}
}
}
return resKibana.Path
}
func getCsvFile(urlKibanaBase string, pathCsvFile string, wg *sync.WaitGroup) error {
defer wg.Done()
res, err := http.Get(urlKibanaBase + pathCsvFile)
if err != nil {
return err
}
defer res.Body.Close()
switch res.StatusCode {
case 200:
dataBody, err := ioutil.ReadAll(res.Body)
if err != nil {
return err
}
err = ioutil.WriteFile(&quot;data.csv&quot;, dataBody, 0666)
if err != nil {
return err
}
return nil
case 503:
fmt.Println(&quot;probably 503/pending&quot;)
return errors.New(&quot;probably 503/pending&quot;)
}
return nil
}

curl request

curl -v localhost:5601/api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u 
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
&gt; GET /api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u HTTP/1.1
&gt; Host: localhost:5601
&gt; User-Agent: curl/7.68.0
&gt; Accept: */*
&gt; 
* Mark bundle as not supporting multiuse
&lt; HTTP/1.1 200 OK
&lt; kbn-csv-contains-formulas: true
&lt; kbn-max-size-reached: false
&lt; content-disposition: inline; filename=&quot;Discover search [2021-09-27T09:19:44.977+02:00].csv&quot;
&lt; content-type: text/csv; charset=utf-8
&lt; x-content-type-options: nosniff
&lt; referrer-policy: no-referrer-when-downgrade
&lt; kbn-name: 3ae0cbefece4
&lt; kbn-license-sig: 58d9fcb437ac7d3ac54d538e6d5ff9a039fde738ed3a940fa530e6d8b6ef6740
&lt; cache-control: private, no-cache, no-store, must-revalidate
&lt; content-length: 9013976
&lt; vary: accept-encoding
&lt; accept-ranges: bytes
&lt; Date: Wed, 29 Sep 2021 14:06:19 GMT
&lt; Connection: keep-alive
&lt; Keep-Alive: timeout=120
&lt; 
{ [13865 bytes data]
100 8802k  100 8802k    0     0  57.6M      0 --:--:-- --:--:-- --:--:-- 57.6M
* Connection #0 to host localhost left intact

<br/>Screenshot that shows that i can get data by typing url path to the browser
Golang程序在进行Get请求后返回HTTP状态码503或待处理。

Update:
<br/> I try to call my function recursively. But it is bad solution even if it's working now.

	case 503:
getCsvFile(urlKibanaBase, pathCsvFile)

答案1

得分: 0

Kibana服务器需要一些时间来准备响应。我已经添加了一些代码来通知通道等待一段时间,然后在等待后重复向客户端发送请求。

如果 res.StatusCode 不等于 200 {
time.Sleep(30 * time.Second)
}
英文:

Kibana Server needs some time to prepare response.
I've added a bit of code for signaling the channel to wait some time and after waiting repeat request to client

if res.StatusCode != 200 {
time.Sleep(30 * time.Second)
}

huangapple
  • 本文由 发表于 2021年9月29日 18:09:57
  • 转载请务必保留本文链接:https://go.coder-hub.com/69374341.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定