英文:
How to design goroutines program to handle api limit error
问题
刚开始学习关于goroutines的强大之处。
我有大约100个账户和10个地区，通过循环遍历它们，使用golang创建大约1000个goroutines来增加读取速度。它运行得太快了，以至于达到了每秒20次的API返回限制。
我该如何确保所有的goroutines都能以最大的调用速率（20/s）维持？我不确定哪种golang并发方法最适合一起处理错误。
例如：
func readInstance(acc string, region string, wg *sync.WaitGroup) {defer wg.Done()response, err := client.DescribeInstances(acc, region)if err != nil {log.Println(err) // API limit exceeding 20return}....}func main() {accounts := []string{"g", "h", "i", ...}regions := []string{"g", "h", "i", ...}for _, region := range regions {for i := 0; i < len(accounts); i++ {wg.Add(1)go readInstance(accounts[i], region, &wg)}}wg.Wait()}
英文:
Just started learning about the power of goroutines.
I have ~100 accounts and ~10 regions, looping through them to create ~ 1000 goroutines with golang to increase the reading speed. It worked too fast that it hit the API return limit of 20/ sec.
How do I ensure that all the goroutines can maintain at the maximum call rate of (20/s)? Im unsure of which golang concurrency methods works best together to handle the error.
eg:
func readInstance(acc string, region string, wg *sync.WaitGroup) {defer wg.Done()response, err := client.DescribeInstances(acc, region)if err != nil {log.Println(err) // API limit exceeding 20return}....}func main() {accounts := []string{"g", "h", "i", ...}regions := []string{"g", "h", "i", ...}for _, region := range regions {for i := 0; i < len(accounts); i++ {wg.Add(1)go readInstance(accounts[i], region, &wg)}}wg.Wait()}
答案1
得分: 3
如果您对在特定实际时间内可以进行的请求数量有一个固定的上限，您可以使用time.NewTicker()来间隔请求。
c := time.NewTicker(50 * time.Millisecond)defer c.Stop()
现在，当您想要进行服务器请求时，只需在实际请求之前插入
<- c.C
即可。
英文:
If you have a fixed upper limit on how many requests you can do in a particular amount of real time, you can use a time.NewTicker() to space things out.
c := time.NewTicker(50 * time.Millisecond)defer c.Stop()
Now, when you want to make a server request, just insert
<- c.C
prior to the actual request.
答案2
得分: 0
我认为你可以尝试使用这个：https://github.com/uber-go/ratelimit
根据文档，它是并发安全的。
import ("fmt""time""go.uber.org/ratelimit")func main() {rl := ratelimit.New(100) // 每秒限制100个请求prev := time.Now()for i := 0; i < 10; i++ {now := rl.Take()fmt.Println(i, now.Sub(prev))prev = now}// 输出:// 0 0// 1 10ms// 2 10ms// 3 10ms// 4 10ms// 5 10ms// 6 10ms// 7 10ms// 8 10ms// 9 10ms}
英文:
i think you can try this: https://github.com/uber-go/ratelimit
According to the documentation, it is concurrency safe.
import ("fmt""time""go.uber.org/ratelimit")func main() {rl := ratelimit.New(100) // per secondprev := time.Now()for i := 0; i < 10; i++ {now := rl.Take()fmt.Println(i, now.Sub(prev))prev = now}// Output:// 0 0// 1 10ms// 2 10ms// 3 10ms// 4 10ms// 5 10ms// 6 10ms// 7 10ms// 8 10ms// 9 10ms}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论