英文:
Golang program after making Get request returns HTTP Status code 503 or pending
问题
我试图通过发送GET请求从Kibana服务器获取CSV数据。
然后服务器会发送带有数据的响应,我将该数据写入CSV文件并保存在某个位置。
每次CSV文件中只有一行"pending"。
Kibana日志显示状态码503。
但是,如果我在浏览器中输入URL,我可以获取带有正确数据的CSV文件。
可能我需要等待更长时间以获取来自Kibana的响应。
从我的角度来看,问题是这样的:服务器可以发送带有一行"pending"的响应,这意味着它需要更多时间来准备正确的响应。
我尝试增加客户端的超时时间,但也没有起作用。
使用Go协程和通道/等待组的思路是:它们强制GET请求等待获取正确的数据,而不是"pending"和503状态码。
以下是代码示例:
package main
import (
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"log"
"net/http"
"sync"
"time"
)
type KibanaReportResponse struct {
Path string `json:"path"`
}
var urlKibanaBase = "http://localhost:5601"
var urlKibanaPost = urlKibanaBase + "/api/reporting/generate/csv_searchsource?"
var urlParameters = "jobParams=(browserTimezone:Europe/Berlin,columns:!(),objectType:search,searchSource:(fields:!((field:'*',include_unmapped:!t)),index:ec074c00-1f62-11ec-8056-8d208a1f6e77,parent:(filter:!(),index:ec074c00-1f62-11ec-8056-8d208a1f6e77,query:(language:kuery,query:'')),sort:!((_score:desc)),trackTotalHits:!t,version:!t),title:'Discover search [2021-09-27T09:19:44.977+02:00]')"
var urlWithParam = urlKibanaPost + urlParameters
func main() {
var wg sync.WaitGroup
wg.Add(1)
pathCsvFile := getCsvPathFromKibana(urlWithParam)
go getCsvFile(urlKibanaBase, pathCsvFile, &wg)
defer wg.Wait()
}
func getCsvPathFromKibana(urlKib string) string {
resKibana := KibanaReportResponse{}
client := &http.Client{
Timeout: 10 * time.Second,
}
if req, err := http.NewRequest("POST", urlKib, nil); err != nil {
log.Println("给定的方法、URL和可选的正文错误", err)
} else {
req.Header.Add("kbn-xsrf", "true")
if res, err := client.Do(req); err != nil {
log.Println("可能是由于客户端策略或HTTP连接的问题", err)
} else {
if err := json.NewDecoder(res.Body).Decode(&resKibana); err != nil {
log.Println("JSON解码问题\n", err)
} else {
return resKibana.Path
}
}
}
return resKibana.Path
}
func getCsvFile(urlKibanaBase string, pathCsvFile string, wg *sync.WaitGroup) error {
defer wg.Done()
res, err := http.Get(urlKibanaBase + pathCsvFile)
if err != nil {
return err
}
defer res.Body.Close()
switch res.StatusCode {
case 200:
dataBody, err := ioutil.ReadAll(res.Body)
if err != nil {
return err
}
err = ioutil.WriteFile("data.csv", dataBody, 0666)
if err != nil {
return err
}
return nil
case 503:
fmt.Println("可能是503/pending")
return errors.New("可能是503/pending")
}
return nil
}
curl请求示例:
curl -v localhost:5601/api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u
截图显示我可以通过在浏览器中输入URL路径来获取数据。
更新:
我尝试递归调用我的函数,但这是一个不好的解决方案,即使它现在可以工作。
case 503:
getCsvFile(urlKibanaBase, pathCsvFile)
希望对你有帮助!
英文:
I try to get csv data from kibana server by making get request.
After this server send me response with data which i write the csv file and save somewhere.
<br/>Every time csv file with single line "pending" inside.
<br/>Kibana logs shows Status code 503
<br/>But if i put url in browser i can get csv file with correct data.
<br/>Probably i need to wait more for response from kibana.
<br/>From my perspective the problem is next: server can send me the response with single line "pending" which means that it needs more time for preparing right response.
<br/>I tried to increase client time but it doesn't work as well
client := http.Client{
Timeout: 10 * time.Second,
}
The Idea with go routines and channels/wait group was next: they force Get request to wait for getting right data instead of "pending" and 503 status code
import (
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"log"
"net/http"
"sync"
)
type KibanaReportResponse struct {
Path string `json:"path"`
}
var urlKibanaBase = "http://localhost:5601"
var urlKibanaPost = urlKibanaBase + "/api/reporting/generate/csv_searchsource?"
var urlParameters = "jobParams=%28browserTimezone%3AEurope%2FBerlin%2Ccolumns%3A%21%28%29%2CobjectType%3Asearch%2CsearchSource%3A%28fields%3A%21%28%28field%3A%27%2A%27%2Cinclude_unmapped%3Atrue%29%29%2Cindex%3Aec074c00-1f62-11ec-8056-8d208a1f6e77%2Cparent%3A%28filter%3A%21%28%29%2Cindex%3Aec074c00-1f62-11ec-8056-8d208a1f6e77%2Cquery%3A%28language%3Akuery%2Cquery%3A%27%27%29%29%2Csort%3A%21%28%28_score%3Adesc%29%29%2CtrackTotalHits%3A%21t%2Cversion%3A%21t%29%2Ctitle%3A%27Discover%20search%20%5B2021-09-27T09%3A19%3A44.977%2B02%3A00%5D%27%29"
var urlWithParam = urlKibanaPost + urlParameters
func main() {
var wg sync.WaitGroup
wg.Add(1)
pathCsvFile := getCsvPathFromKibana(urlWithParam)
go getCsvFile(urlKibanaBase, pathCsvFile, &wg)
defer wg.Wait()
}
func getCsvPathFromKibana(urlKib string) string {
resKibana := KibanaReportResponse{}
client := &http.Client{}
if req, err := http.NewRequest("POST", urlKib, nil); err != nil {
log.Println("Given a method, URL, andoptional body are wrong", err)
} else {
req.Header.Add("kbn-xsrf", "true")
if res, err := client.Do(req); err != nil {
log.Println("Probably problems depends on client policy or HTTP connection", err)
} else {
if err := json.NewDecoder(res.Body).Decode(&resKibana); err != nil {
log.Println("Problem by Json decoding \n", err)
} else {
return resKibana.Path
}
}
}
return resKibana.Path
}
func getCsvFile(urlKibanaBase string, pathCsvFile string, wg *sync.WaitGroup) error {
defer wg.Done()
res, err := http.Get(urlKibanaBase + pathCsvFile)
if err != nil {
return err
}
defer res.Body.Close()
switch res.StatusCode {
case 200:
dataBody, err := ioutil.ReadAll(res.Body)
if err != nil {
return err
}
err = ioutil.WriteFile("data.csv", dataBody, 0666)
if err != nil {
return err
}
return nil
case 503:
fmt.Println("probably 503/pending")
return errors.New("probably 503/pending")
}
return nil
}
curl request
curl -v localhost:5601/api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
> GET /api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u HTTP/1.1
> Host: localhost:5601
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< kbn-csv-contains-formulas: true
< kbn-max-size-reached: false
< content-disposition: inline; filename="Discover search [2021-09-27T09:19:44.977+02:00].csv"
< content-type: text/csv; charset=utf-8
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< kbn-name: 3ae0cbefece4
< kbn-license-sig: 58d9fcb437ac7d3ac54d538e6d5ff9a039fde738ed3a940fa530e6d8b6ef6740
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 9013976
< vary: accept-encoding
< accept-ranges: bytes
< Date: Wed, 29 Sep 2021 14:06:19 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
<
{ [13865 bytes data]
100 8802k 100 8802k 0 0 57.6M 0 --:--:-- --:--:-- --:--:-- 57.6M
* Connection #0 to host localhost left intact
<br/>Screenshot that shows that i can get data by typing url path to the browser
Update:
<br/> I try to call my function recursively. But it is bad solution even if it's working now.
case 503:
getCsvFile(urlKibanaBase, pathCsvFile)
答案1
得分: 0
Kibana服务器需要一些时间来准备响应。我已经添加了一些代码来通知通道等待一段时间,然后在等待后重复向客户端发送请求。
如果 res.StatusCode 不等于 200 {
time.Sleep(30 * time.Second)
}
英文:
Kibana Server needs some time to prepare response.
I've added a bit of code for signaling the channel to wait some time and after waiting repeat request to client
if res.StatusCode != 200 {
time.Sleep(30 * time.Second)
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论