Golang程序在进行Get请求后返回HTTP状态码503或待处理。

huangapple go评论142阅读模式
英文:

Golang program after making Get request returns HTTP Status code 503 or pending

问题

我试图通过发送GET请求从Kibana服务器获取CSV数据。
然后服务器会发送带有数据的响应,我将该数据写入CSV文件并保存在某个位置。
每次CSV文件中只有一行"pending"。
Kibana日志显示状态码503。
但是,如果我在浏览器中输入URL,我可以获取带有正确数据的CSV文件。
可能我需要等待更长时间以获取来自Kibana的响应。
从我的角度来看,问题是这样的:服务器可以发送带有一行"pending"的响应,这意味着它需要更多时间来准备正确的响应。
我尝试增加客户端的超时时间,但也没有起作用。

使用Go协程和通道/等待组的思路是:它们强制GET请求等待获取正确的数据,而不是"pending"和503状态码。

以下是代码示例:

  1. package main
  2. import (
  3. "encoding/json"
  4. "errors"
  5. "fmt"
  6. "io/ioutil"
  7. "log"
  8. "net/http"
  9. "sync"
  10. "time"
  11. )
  12. type KibanaReportResponse struct {
  13. Path string `json:"path"`
  14. }
  15. var urlKibanaBase = "http://localhost:5601"
  16. var urlKibanaPost = urlKibanaBase + "/api/reporting/generate/csv_searchsource?"
  17. var urlParameters = "jobParams=(browserTimezone:Europe/Berlin,columns:!(),objectType:search,searchSource:(fields:!((field:'*',include_unmapped:!t)),index:ec074c00-1f62-11ec-8056-8d208a1f6e77,parent:(filter:!(),index:ec074c00-1f62-11ec-8056-8d208a1f6e77,query:(language:kuery,query:'')),sort:!((_score:desc)),trackTotalHits:!t,version:!t),title:'Discover search [2021-09-27T09:19:44.977+02:00]')"
  18. var urlWithParam = urlKibanaPost + urlParameters
  19. func main() {
  20. var wg sync.WaitGroup
  21. wg.Add(1)
  22. pathCsvFile := getCsvPathFromKibana(urlWithParam)
  23. go getCsvFile(urlKibanaBase, pathCsvFile, &wg)
  24. defer wg.Wait()
  25. }
  26. func getCsvPathFromKibana(urlKib string) string {
  27. resKibana := KibanaReportResponse{}
  28. client := &http.Client{
  29. Timeout: 10 * time.Second,
  30. }
  31. if req, err := http.NewRequest("POST", urlKib, nil); err != nil {
  32. log.Println("给定的方法、URL和可选的正文错误", err)
  33. } else {
  34. req.Header.Add("kbn-xsrf", "true")
  35. if res, err := client.Do(req); err != nil {
  36. log.Println("可能是由于客户端策略或HTTP连接的问题", err)
  37. } else {
  38. if err := json.NewDecoder(res.Body).Decode(&resKibana); err != nil {
  39. log.Println("JSON解码问题\n", err)
  40. } else {
  41. return resKibana.Path
  42. }
  43. }
  44. }
  45. return resKibana.Path
  46. }
  47. func getCsvFile(urlKibanaBase string, pathCsvFile string, wg *sync.WaitGroup) error {
  48. defer wg.Done()
  49. res, err := http.Get(urlKibanaBase + pathCsvFile)
  50. if err != nil {
  51. return err
  52. }
  53. defer res.Body.Close()
  54. switch res.StatusCode {
  55. case 200:
  56. dataBody, err := ioutil.ReadAll(res.Body)
  57. if err != nil {
  58. return err
  59. }
  60. err = ioutil.WriteFile("data.csv", dataBody, 0666)
  61. if err != nil {
  62. return err
  63. }
  64. return nil
  65. case 503:
  66. fmt.Println("可能是503/pending")
  67. return errors.New("可能是503/pending")
  68. }
  69. return nil
  70. }

curl请求示例:

  1. curl -v localhost:5601/api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u

截图显示我可以通过在浏览器中输入URL路径来获取数据。

更新:
我尝试递归调用我的函数,但这是一个不好的解决方案,即使它现在可以工作。

  1. case 503:
  2. getCsvFile(urlKibanaBase, pathCsvFile)

希望对你有帮助!

英文:

I try to get csv data from kibana server by making get request.
After this server send me response with data which i write the csv file and save somewhere.
<br/>Every time csv file with single line "pending" inside.
<br/>Kibana logs shows Status code 503
<br/>But if i put url in browser i can get csv file with correct data.
<br/>Probably i need to wait more for response from kibana.
<br/>From my perspective the problem is next: server can send me the response with single line "pending" which means that it needs more time for preparing right response.
<br/>I tried to increase client time but it doesn't work as well

  1. client := http.Client{
  2. Timeout: 10 * time.Second,
  3. }

The Idea with go routines and channels/wait group was next: they force Get request to wait for getting right data instead of "pending" and 503 status code

  1. import (
  2. &quot;encoding/json&quot;
  3. &quot;errors&quot;
  4. &quot;fmt&quot;
  5. &quot;io/ioutil&quot;
  6. &quot;log&quot;
  7. &quot;net/http&quot;
  8. &quot;sync&quot;
  9. )
  10. type KibanaReportResponse struct {
  11. Path string `json:&quot;path&quot;`
  12. }
  13. var urlKibanaBase = &quot;http://localhost:5601&quot;
  14. var urlKibanaPost = urlKibanaBase + &quot;/api/reporting/generate/csv_searchsource?&quot;
  15. var urlParameters = &quot;jobParams=%28browserTimezone%3AEurope%2FBerlin%2Ccolumns%3A%21%28%29%2CobjectType%3Asearch%2CsearchSource%3A%28fields%3A%21%28%28field%3A%27%2A%27%2Cinclude_unmapped%3Atrue%29%29%2Cindex%3Aec074c00-1f62-11ec-8056-8d208a1f6e77%2Cparent%3A%28filter%3A%21%28%29%2Cindex%3Aec074c00-1f62-11ec-8056-8d208a1f6e77%2Cquery%3A%28language%3Akuery%2Cquery%3A%27%27%29%29%2Csort%3A%21%28%28_score%3Adesc%29%29%2CtrackTotalHits%3A%21t%2Cversion%3A%21t%29%2Ctitle%3A%27Discover%20search%20%5B2021-09-27T09%3A19%3A44.977%2B02%3A00%5D%27%29&quot;
  16. var urlWithParam = urlKibanaPost + urlParameters
  17. func main() {
  18. var wg sync.WaitGroup
  19. wg.Add(1)
  20. pathCsvFile := getCsvPathFromKibana(urlWithParam)
  21. go getCsvFile(urlKibanaBase, pathCsvFile, &amp;wg)
  22. defer wg.Wait()
  23. }
  24. func getCsvPathFromKibana(urlKib string) string {
  25. resKibana := KibanaReportResponse{}
  26. client := &amp;http.Client{}
  27. if req, err := http.NewRequest(&quot;POST&quot;, urlKib, nil); err != nil {
  28. log.Println(&quot;Given a method, URL, andoptional body are wrong&quot;, err)
  29. } else {
  30. req.Header.Add(&quot;kbn-xsrf&quot;, &quot;true&quot;)
  31. if res, err := client.Do(req); err != nil {
  32. log.Println(&quot;Probably problems depends on client policy or HTTP connection&quot;, err)
  33. } else {
  34. if err := json.NewDecoder(res.Body).Decode(&amp;resKibana); err != nil {
  35. log.Println(&quot;Problem by Json decoding \n&quot;, err)
  36. } else {
  37. return resKibana.Path
  38. }
  39. }
  40. }
  41. return resKibana.Path
  42. }
  43. func getCsvFile(urlKibanaBase string, pathCsvFile string, wg *sync.WaitGroup) error {
  44. defer wg.Done()
  45. res, err := http.Get(urlKibanaBase + pathCsvFile)
  46. if err != nil {
  47. return err
  48. }
  49. defer res.Body.Close()
  50. switch res.StatusCode {
  51. case 200:
  52. dataBody, err := ioutil.ReadAll(res.Body)
  53. if err != nil {
  54. return err
  55. }
  56. err = ioutil.WriteFile(&quot;data.csv&quot;, dataBody, 0666)
  57. if err != nil {
  58. return err
  59. }
  60. return nil
  61. case 503:
  62. fmt.Println(&quot;probably 503/pending&quot;)
  63. return errors.New(&quot;probably 503/pending&quot;)
  64. }
  65. return nil
  66. }

curl request

  1. curl -v localhost:5601/api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u
  1. * TCP_NODELAY set
  2. * Connected to localhost (127.0.0.1) port 5601 (#0)
  3. &gt; GET /api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u HTTP/1.1
  4. &gt; Host: localhost:5601
  5. &gt; User-Agent: curl/7.68.0
  6. &gt; Accept: */*
  7. &gt;
  8. * Mark bundle as not supporting multiuse
  9. &lt; HTTP/1.1 200 OK
  10. &lt; kbn-csv-contains-formulas: true
  11. &lt; kbn-max-size-reached: false
  12. &lt; content-disposition: inline; filename=&quot;Discover search [2021-09-27T09:19:44.977+02:00].csv&quot;
  13. &lt; content-type: text/csv; charset=utf-8
  14. &lt; x-content-type-options: nosniff
  15. &lt; referrer-policy: no-referrer-when-downgrade
  16. &lt; kbn-name: 3ae0cbefece4
  17. &lt; kbn-license-sig: 58d9fcb437ac7d3ac54d538e6d5ff9a039fde738ed3a940fa530e6d8b6ef6740
  18. &lt; cache-control: private, no-cache, no-store, must-revalidate
  19. &lt; content-length: 9013976
  20. &lt; vary: accept-encoding
  21. &lt; accept-ranges: bytes
  22. &lt; Date: Wed, 29 Sep 2021 14:06:19 GMT
  23. &lt; Connection: keep-alive
  24. &lt; Keep-Alive: timeout=120
  25. &lt;
  26. { [13865 bytes data]
  27. 100 8802k 100 8802k 0 0 57.6M 0 --:--:-- --:--:-- --:--:-- 57.6M
  28. * Connection #0 to host localhost left intact

<br/>Screenshot that shows that i can get data by typing url path to the browser
Golang程序在进行Get请求后返回HTTP状态码503或待处理。

Update:
<br/> I try to call my function recursively. But it is bad solution even if it's working now.

  1. case 503:
  2. getCsvFile(urlKibanaBase, pathCsvFile)

答案1

得分: 0

Kibana服务器需要一些时间来准备响应。我已经添加了一些代码来通知通道等待一段时间,然后在等待后重复向客户端发送请求。

  1. 如果 res.StatusCode 不等于 200 {
  2. time.Sleep(30 * time.Second)
  3. }
英文:

Kibana Server needs some time to prepare response.
I've added a bit of code for signaling the channel to wait some time and after waiting repeat request to client

  1. if res.StatusCode != 200 {
  2. time.Sleep(30 * time.Second)
  3. }

huangapple
  • 本文由 发表于 2021年9月29日 18:09:57
  • 转载请务必保留本文链接:https://go.coder-hub.com/69374341.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定