消费分页的REST API

huangapple go评论139阅读模式
英文:

Consuming Paginated REST API

问题

我正在尝试使用一个API,每次成功请求后都会返回以下json键:

  1. {
  2. "hasNextPage": true,
  3. "endCursor": "some_guid_present_here"
  4. }

API的工作方式是检查是否有下一页,然后使用适当的游标继续分页。

然而,每次使用这个结构时,即使response.Paginations.HasNextPage为false,它有时仍会保持循环。

我正在尝试理解是否是我以这种方式消耗分页API的结构导致了这个问题。

此外,我有4-5个起始请求,我通过goroutines分别发送。我不确定这是否会引起问题,但我也在etl.go中附上了这个。

主要的请求结构在api.go中。

我已经确认我收到了响应,并且它们正确地解组,但我正在努力解决这种不确定的行为。

api.go

  1. package models
  2. import (
  3. "encoding/json"
  4. "io/ioutil"
  5. "net/http"
  6. "fmt"
  7. )
  8. type Request struct {
  9. Url string
  10. ApiKey string
  11. }
  12. type Response struct {
  13. ...一些字段...
  14. Paginations Pagination `json:"pagination"`
  15. }
  16. type Pagination struct {
  17. EndCursor string `json:"endCursor"`
  18. HasNextPage bool `json:"hasNextPage"`
  19. }
  20. func (request *Request) Get() ([]Response, error) {
  21. var responses []Response
  22. var response Response
  23. // 设置新请求
  24. req, err := http.NewRequest("GET", request.Url, nil)
  25. if err != nil {
  26. fmt.Println("创建请求时出错...")
  27. return responses, err
  28. }
  29. // 添加请求头
  30. req.Header = http.Header{
  31. "accept": {"application/json"},
  32. "authorization": {"Bearer " + request.ApiKey},
  33. }
  34. // 从API获取初始响应并捕获状态码
  35. resp, _ := http.DefaultClient.Do(req)
  36. response.Status = resp.StatusCode
  37. // 读取响应体并解组为结构体
  38. respBody, err := ioutil.ReadAll(resp.Body)
  39. json.Unmarshal(respBody, &response)
  40. // 如果有解析错误,记录下来
  41. if err != nil {
  42. fmt.Println(err)
  43. }
  44. defer resp.Body.Close()
  45. // 这个字段将在响应有效负载中
  46. // 它已验证为bool类型(而不是字符串)
  47. fmt.Printf("是否有下一页?%t\n", response.Paginations.HasNextPage)
  48. // 将响应追加到响应列表中
  49. responses = append(responses, response)
  50. // 如果有分页,继续循环直到所有分页都用完
  51. for response.Paginations.HasNextPage == true {
  52. req, err := http.NewRequest("GET", request.Url, nil)
  53. if err != nil {
  54. return responses, err
  55. }
  56. // 在查询中添加“after”游标,以便重定向到分页响应
  57. qry := req.URL.Query()
  58. qry.Set("after", response.Paginations.EndCursor)
  59. req.URL.RawQuery = qry.Encode()
  60. fmt.Println("分页请求查询:", req.URL.String())
  61. // 发送请求
  62. resp, err := http.DefaultClient.Do(req)
  63. response.Status = resp.StatusCode
  64. fmt.Printf("状态码:%d\n", response.Status)
  65. // 读取响应并反序列化
  66. respBody, err := ioutil.ReadAll(resp.Body)
  67. json.Unmarshal(respBody, &response)
  68. fmt.Println("分页游标:", response.Paginations.EndCursor)
  69. fmt.Printf("是否有下一页?%t\n", response.Paginations.HasNextPage)
  70. // 如果有解析错误,记录下来
  71. if err != nil {
  72. fmt.Println(err)
  73. }
  74. defer resp.Body.Close()
  75. // 将响应追加到响应列表中
  76. responses = append(responses, response)
  77. }
  78. return responses, nil
  79. }

etl.go

  1. package models
  2. import (
  3. "fmt"
  4. "sync"
  5. )
  6. type Etl struct {
  7. Requests []Request
  8. }
  9. func (etl *Etl) Extract() [][]Response {
  10. var wg sync.WaitGroup
  11. ch := make(chan []Response)
  12. defer close(ch)
  13. for _, req := range etl.Requests {
  14. wg.Add(1) // 增加计数
  15. fmt.Println("增加等待组")
  16. go func(i Request) {
  17. defer wg.Done() // 减少计数
  18. resp, err := req.Get()
  19. if err != nil {
  20. fmt.Println(err)
  21. }
  22. ch <- resp
  23. fmt.Println("减少等待组")
  24. }(req)
  25. }
  26. results := make([][]Response, len(etl.Requests))
  27. for i, _ := range results {
  28. results[i] = <-ch
  29. //fmt.Println(results[i])
  30. }
  31. wg.Wait()
  32. return nil
  33. }

希望对你有所帮助!

英文:

I am trying to consume an API that on each successful request has a json key like the following:

  1. {
  2. &quot;hasNextPage&quot;: true,
  3. &quot;endCursor&quot;: &quot;some_guid_present_here&quot;
  4. }

The way the API works (I have used it many times in Python but am trying with Go for separate use case) is to basically check if there is a next page, then use the appropriate cursor to continue to paginate.

However, every time I use this structure, it will sometimes keep looping even though the response.Paginations.HasNextPage will be false.

I am trying to understand if it is my structure of consuming a paginated API like this that is causing this or else.

Also, I have say 4-5 requests to start off with, which I sent separately via goroutines. I am not sure if this is causing an issue but I've attached that as well in etl.go.

The main request making structs are in api.go.

I've confirmed that I do receive responses and they are unmarshaling properly, but am trying to wrangle this uncertain behavior.

api.go

  1. package models
  2. import (
  3. &quot;encoding/json&quot;
  4. &quot;io/ioutil&quot;
  5. &quot;net/http&quot;
  6. &quot;fmt&quot;
  7. )
  8. type Request struct {
  9. Url string
  10. ApiKey string
  11. }
  12. type Response struct {
  13. ...some fields...
  14. Paginations Pagination `json:&quot;pagination&quot;`
  15. }
  16. type Pagination struct {
  17. EndCursor string `json:&quot;endCursor&quot;`
  18. HasNextPage bool `json:&quot;hasNextPage&quot;`
  19. }
  20. func (request *Request) Get() ([]Response, error) {
  21. var responses []Response
  22. var response Response
  23. // Set up new request
  24. req, err := http.NewRequest(&quot;GET&quot;, request.Url, nil)
  25. if err != nil {
  26. fmt.Println(&quot;Error creating request...&quot;)
  27. return responses, err
  28. }
  29. // Add request headers
  30. req.Header = http.Header{
  31. &quot;accept&quot;: {&quot;application/json&quot;},
  32. &quot;authorization&quot;: {&quot;Bearer &quot; + request.ApiKey},
  33. }
  34. // Get our initial response from the API and capture status code
  35. resp, _ := http.DefaultClient.Do(req)
  36. response.Status = resp.StatusCode
  37. // Read the response body and Unmarshal into struct
  38. respBody, err := ioutil.ReadAll(resp.Body)
  39. json.Unmarshal(respBody, &amp;response)
  40. // If there was a parsing error, log it
  41. if err != nil {
  42. fmt.Println(err)
  43. }
  44. defer resp.Body.Close()
  45. // This field will be in the response payload
  46. // It is verified to be of type bool (not string)
  47. fmt.Printf(&quot;Has Next Page? %t\n&quot;, resp.Paginations.HasNextPage)
  48. // Append response to our slice of responses
  49. responses = append(responses, response)
  50. // If paginations are available, continue to loop through until all paginations are exhausted
  51. for response.Paginations.HasNextPage == true {
  52. req, err := http.NewRequest(&quot;GET&quot;, request.Url, nil)
  53. if err != nil {
  54. return responses, err
  55. }
  56. // Append &quot;after&quot; cursor to query in order to redirect to paginated response
  57. qry := req.URL.Query()
  58. qry.Set(&quot;after&quot;, response.Paginations.EndCursor)
  59. req.URL.RawQuery = qry.Encode()
  60. fmt.Println(&quot;Paginated request query: &quot;, req.URL.String())
  61. // Make request
  62. resp, err := http.DefaultClient.Do(req)
  63. response.Status = resp.StatusCode
  64. fmt.Printf(&quot;Status Code: %d\n&quot;, response.Status)
  65. // Read response and deserialize it
  66. respBody, err := ioutil.ReadAll(resp.Body)
  67. json.Unmarshal(respBody, &amp;response)
  68. fmt.Println(&quot;Pagination Cursor: &quot;, response.Paginations.EndCursor)
  69. fmt.Printf(&quot;Has Next Page? %t\n&quot;, response.Paginations.HasNextPage)
  70. // If there was a parsing error, log it
  71. if err != nil {
  72. fmt.Println(err)
  73. }
  74. defer resp.Body.Close()
  75. // Append response to our slice of responses
  76. responses = append(responses, response)
  77. }
  78. return responses, nil
  79. }

etl.go

  1. package models
  2. import (
  3. &quot;fmt&quot;
  4. &quot;sync&quot;
  5. )
  6. type Etl struct {
  7. Requests []Request
  8. }
  9. func (etl *Etl) Extract() [][]Response {
  10. var wg sync.WaitGroup
  11. ch := make(chan []Response)
  12. defer close(ch)
  13. for _, req := range etl.Requests {
  14. wg.Add(1) // Increment
  15. fmt.Println(&quot;Incremented wait group&quot;)
  16. go func(i Request) {
  17. defer wg.Done() // Decrement
  18. resp, err := req.Get()
  19. if err != nil {
  20. fmt.Println(err)
  21. }
  22. ch &lt;- resp
  23. fmt.Println(&quot;Decremented wait group&quot;)
  24. }(req)
  25. }
  26. results := make([][]Response, len(etl.Requests))
  27. for i, _ := range results {
  28. results[i] = &lt;-ch
  29. //fmt.Println(results[i])
  30. }
  31. wg.Wait()
  32. return nil
  33. }

答案1

得分: 1

我相信我找到了问题所在。在我的分页循环中,for response.Paginations.HasNextPage == true,我在每次迭代中创建了一个新的请求对象(http.NewRequest),该对象没有添加前一个(初始请求)的标头。

这导致返回401未经授权错误并持续查询API,因为它没有接收到新的response.Paginations.HasNextPage值。

我的解决方案是稍微修改for循环,如下所示:

  1. package models
  2. import (
  3. "encoding/json"
  4. "io/ioutil"
  5. "net/http"
  6. "fmt"
  7. )
  8. type Request struct {
  9. Url string
  10. ApiKey string
  11. }
  12. type Response struct {
  13. // ...一些字段...
  14. Paginations Pagination `json:"pagination"`
  15. }
  16. type Pagination struct {
  17. EndCursor string `json:"endCursor"`
  18. HasNextPage bool `json:"hasNextPage"`
  19. }
  20. func (request *Request) Get() ([]Response, error) {
  21. var responses []Response
  22. var response Response
  23. // 设置新的请求
  24. req, err := http.NewRequest("GET", request.Url, nil)
  25. if err != nil {
  26. fmt.Println("创建请求时出错...")
  27. return responses, err
  28. }
  29. // 添加请求标头
  30. req.Header = http.Header{
  31. "accept": {"application/json"},
  32. "authorization": {"Bearer " + request.ApiKey},
  33. }
  34. // 从API获取初始响应并捕获状态码
  35. resp, _ := http.DefaultClient.Do(req)
  36. response.Status = resp.StatusCode
  37. // 读取响应正文并解组为结构体
  38. respBody, err := ioutil.ReadAll(resp.Body)
  39. json.Unmarshal(respBody, &response)
  40. // 如果有解析错误,记录下来
  41. if err != nil {
  42. fmt.Println(err)
  43. }
  44. defer resp.Body.Close()
  45. // 这个字段将在响应有效负载中
  46. // 它被验证为bool类型(而不是字符串)
  47. fmt.Printf("是否有下一页?%t\n", response.Paginations.HasNextPage)
  48. // 将响应附加到我们的响应切片中
  49. responses = append(responses, response)
  50. // 如果有分页可用,继续循环直到所有分页用尽
  51. for response.Paginations.HasNextPage == true {
  52. // 将“after”游标附加到查询中,以便重定向到分页响应
  53. qry := req.URL.Query()
  54. qry.Set("after", response.Paginations.EndCursor)
  55. req.URL.RawQuery = qry.Encode()
  56. fmt.Println("分页请求查询:", req.URL.String())
  57. // 发送请求
  58. resp, err := http.DefaultClient.Do(req)
  59. response.Status = resp.StatusCode
  60. fmt.Printf("状态码:%d\n", response.Status)
  61. // 读取响应并反序列化
  62. respBody, err := ioutil.ReadAll(resp.Body)
  63. json.Unmarshal(respBody, &response)
  64. fmt.Println("分页游标:", response.Paginations.EndCursor)
  65. fmt.Printf("是否有下一页?%t\n", response.Paginations.HasNextPage)
  66. // 如果有解析错误,记录下来
  67. if err != nil {
  68. fmt.Println(err)
  69. }
  70. defer resp.Body.Close()
  71. // 将响应附加到我们的响应切片中
  72. responses = append(responses, response)
  73. }
  74. return responses, nil
  75. }

以上是翻译好的内容,请查阅。

英文:

I believe I found the issue. In my pagination loop for response.Paginations.HasNextPage == true I was creating a new request object (http.NewRequest) on each iteration which did not have headers added from the prior (initial request).

This caused a 401 unauthorized error to return and continuous querying of the API since it was not receiving a new response.Paginations.HasNextPage value.

My solution was to simply alter the for loop a bit like so:

  1. package models
  2. import (
  3. &quot;encoding/json&quot;
  4. &quot;io/ioutil&quot;
  5. &quot;net/http&quot;
  6. &quot;fmt&quot;
  7. )
  8. type Request struct {
  9. Url string
  10. ApiKey string
  11. }
  12. type Response struct {
  13. ...some fields...
  14. Paginations Pagination `json:&quot;pagination&quot;`
  15. }
  16. type Pagination struct {
  17. EndCursor string `json:&quot;endCursor&quot;`
  18. HasNextPage bool `json:&quot;hasNextPage&quot;`
  19. }
  20. func (request *Request) Get() ([]Response, error) {
  21. var responses []Response
  22. var response Response
  23. // Set up new request
  24. req, err := http.NewRequest(&quot;GET&quot;, request.Url, nil)
  25. if err != nil {
  26. fmt.Println(&quot;Error creating request...&quot;)
  27. return responses, err
  28. }
  29. // Add request headers
  30. req.Header = http.Header{
  31. &quot;accept&quot;: {&quot;application/json&quot;},
  32. &quot;authorization&quot;: {&quot;Bearer &quot; + request.ApiKey},
  33. }
  34. // Get our initial response from the API and capture status code
  35. resp, _ := http.DefaultClient.Do(req)
  36. response.Status = resp.StatusCode
  37. // Read the response body and Unmarshal into struct
  38. respBody, err := ioutil.ReadAll(resp.Body)
  39. json.Unmarshal(respBody, &amp;response)
  40. // If there was a parsing error, log it
  41. if err != nil {
  42. fmt.Println(err)
  43. }
  44. defer resp.Body.Close()
  45. // This field will be in the response payload
  46. // It is verified to be of type bool (not string)
  47. fmt.Printf(&quot;Has Next Page? %t\n&quot;, resp.Paginations.HasNextPage)
  48. // Append response to our slice of responses
  49. responses = append(responses, response)
  50. // If paginations are available, continue to loop through until all paginations are exhausted
  51. for response.Paginations.HasNextPage == true {
  52. // Append &quot;after&quot; cursor to query in order to redirect to paginated response
  53. qry := req.URL.Query()
  54. qry.Set(&quot;after&quot;, response.Paginations.EndCursor)
  55. req.URL.RawQuery = qry.Encode()
  56. fmt.Println(&quot;Paginated request query: &quot;, req.URL.String())
  57. // Make request
  58. resp, err := http.DefaultClient.Do(req)
  59. response.Status = resp.StatusCode
  60. fmt.Printf(&quot;Status Code: %d\n&quot;, response.Status)
  61. // Read response and deserialize it
  62. respBody, err := ioutil.ReadAll(resp.Body)
  63. json.Unmarshal(respBody, &amp;response)
  64. fmt.Println(&quot;Pagination Cursor: &quot;, response.Paginations.EndCursor)
  65. fmt.Printf(&quot;Has Next Page? %t\n&quot;, response.Paginations.HasNextPage)
  66. // If there was a parsing error, log it
  67. if err != nil {
  68. fmt.Println(err)
  69. }
  70. defer resp.Body.Close()
  71. // Append response to our slice of responses
  72. responses = append(responses, response)
  73. }
  74. return responses, nil
  75. }

huangapple
  • 本文由 发表于 2023年1月25日 04:41:35
  • 转载请务必保留本文链接:https://go.coder-hub.com/75227071.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定