checking api with golang for changes in data continuously

I'm trying to poll an API to keep a time series of traffic data, and save that data to postgres when there has been a change.

At the moment I've got an implementation sort of like this

//this needs to check the api for new information every X seconds
func Poll(req *http.Request, client *http.Client) ([]byte, error) {
	r := rand.New(rand.NewSource(99))
	c := time.Tick(10 * time.Second)
	for _ = range c {
		//Download the current contents of the URL and do something with it
		response, err := client.Do(req)
		data, _ := io.ReadAll(response.Body)

		if err != nil {
			return nil, err
		}
		return data, nil
		// add a bit of jitter
		jitter := time.Duration(r.Int31n(5000)) * time.Millisecond
		time.Sleep(jitter)
	}

}



func main() {

	client := &http.Client{
		Timeout: time.Second * 60 * 60 * 600,
	}
	url := "https://data-exchange-api.vicroads.vic.gov.au/bluetooth_data/links"
	req, err := http.NewRequest("GET", url, nil)
	if err != nil {
		return err
	}
	req.Header.Set("Ocp-Apim-Subscription-Key", "xx")

	// response, err := client.Do(req)
	data, err := Poll(req, client)
	fmt.Println(string(data))

}

I will do a comparison function next.

Basically, I'm trying to work out how to ensure the loop calls the query in the first place and returns an appropriate value.

I think this implementation is probably not very good and I'm just not sure how to really properly implement it. Could I get some pointers?

Your problem presents a typical producer/consumer scenario since your Poll() function is producing the response data which is consumed by your main() function (may be to save data in postgres).
This problem can be solved excellently by using go routines and channels.

The polling work can be done in a goroutine which communicates the response data to the main function over a channel. There could also be an error while the polling work (response error or io error) so it should also be communicated to the main() function.

First define a new type to hold polled data and an error:

type PollResponse struct {
	Data []byte
	Err error
}

In Poll() function, start a go routine to do poll work and return a channel to share data outside the go routine:

func Poll(req *http.Request, client *http.Client) (ch chan PollResponse){
	ch = make(chan PollResponse) // Buffered channel is also good
	go func() {
		defer func() {
			close(ch)
		}()
		r := rand.New(rand.NewSource(99))
		c := time.Tick(10 * time.Second)

		for range c {
			res, err := client.Do(req);
			pollRes := PollResponse {}
			if err != nil {
				pollRes.Data, pollRes.Err = nil, err
				ch <- pollRes
				break
			}
			pollRes.Data, pollRes.Err = io.ReadAll(res.Body)
			ch <- pollRes
			if pollRes.Err != nil {
				break
			}
			jitter := time.Duration(r.Int31n(5000)) * time.Millisecond
			time.Sleep(jitter)
		}
	}()
	return
}

And finally in the main() function, call Poll() and read the channel to get poll response:

func main() {
	client := &http.Client{
		Timeout: time.Second * 60 * 60 * 600,
	}
	url := "https://data-exchange-api.vicroads.vic.gov.au/bluetooth_data/links"

	req, err := http.NewRequest("GET", url, nil)
	if err != nil {
		return
	}
    req.Header.Set("Ocp-Apim-Subscription-Key", "xx")

	pollCh := Poll(req, client)
	
	for item := range pollCh {
		if item.Err == nil {
			fmt.Println(string(item.Data)) // or save it to postgres database
		}		
	}
}

Range over the ticker channel. On each iteration, get the data, check if the data has changed and process the data. The key point is to process the data from inside of loop instead of returning the data from the function.

Assuming that you have the following function:

// procesChangedData updates the database with new
// data from the API endpoint.
func processChangedData(data []byte) error {
    // implement save to postgress
}

Use the following function to poll:

func Poll(client *http.Client) error {

	url := "https://data-exchange-api.vicroads.vic.gov.au/bluetooth_data/links"

	// Use NewTicker instead of Tick so we can cleanup
	// ticker on return from the function.
	t := time.NewTicker(10 * time.Second)
	defer t.Stop()

	var prev []byte

	for _ = range t.C {

		// Create a new request objet for each request.
		req, err := http.NewRequest("GET", url, nil)
		if err != nil {
			return err
		}
		req.Header.Set("Ocp-Apim-Subscription-Key", "xx")

		resp, err := client.Do(req)
		if err != nil {
            // Edit error handling to match application 
            // requirements. I return an error here. Continuing
            // the loop is also an option.
            return err
		}

		data, err := io.ReadAll(resp.Body)

		// Ensure that body is closed before handling errors
		// below.
		resp.Body.Close()

		if err != nil {
            // Edit error handling to match application 
            // requirements. I return an error here. Continuing
            // the loop is also an option.
            return err
		}

		if resp.StatusCode != http.StatusOK {
            // Edit error handling to match application 
            // requirements. I return an error here. Continuing
            // the loop is also an option.
            return fmt.Errorf("bad status %d", resp.StatusCode)
		}

		if bytes.Equal(data, prev) {
			continue
		}
		prev = data

		if err := processChangedData(data); err != nil {
            // Edit error handling to match application 
            // requirements. I return an error here. Continuing
            // the loop is also an option.
            return err
		}
	}
	panic("unexpected break from loop")
}