英文:
How to design goroutines program to handle api limit error
问题
刚开始学习关于goroutines的强大之处。
我有大约100个账户和10个地区，通过循环遍历它们，使用golang创建大约1000个goroutines来增加读取速度。它运行得太快了，以至于达到了每秒20次的API返回限制。
我该如何确保所有的goroutines都能以最大的调用速率（20/s）维持？我不确定哪种golang并发方法最适合一起处理错误。
例如：
func readInstance(acc string, region string, wg *sync.WaitGroup) {
defer wg.Done()
response, err := client.DescribeInstances(acc, region)
if err != nil {
log.Println(err) // API limit exceeding 20
return
}
.
.
.
.
}
func main() {
accounts := []string{"g", "h", "i", ...}
regions := []string{"g", "h", "i", ...}
for _, region := range regions {
for i := 0; i < len(accounts); i++ {
wg.Add(1)
go readInstance(accounts[i], region, &wg)
}
}
wg.Wait()
}
英文:
Just started learning about the power of goroutines.
I have ~100 accounts and ~10 regions, looping through them to create ~ 1000 goroutines with golang to increase the reading speed. It worked too fast that it hit the API return limit of 20/ sec.
How do I ensure that all the goroutines can maintain at the maximum call rate of (20/s)? Im unsure of which golang concurrency methods works best together to handle the error.
eg:
func readInstance(acc string, region string, wg *sync.WaitGroup) {
defer wg.Done()
response, err := client.DescribeInstances(acc, region)
if err != nil {
log.Println(err) // API limit exceeding 20
return
}
.
.
.
.
}
func main() {
accounts := []string{"g", "h", "i", ...}
regions := []string{"g", "h", "i", ...}
for _, region := range regions {
for i := 0; i < len(accounts); i++ {
wg.Add(1)
go readInstance(accounts[i], region, &wg)
}
}
wg.Wait()
}
答案1
得分: 3
如果您对在特定实际时间内可以进行的请求数量有一个固定的上限，您可以使用time.NewTicker()来间隔请求。
c := time.NewTicker(50 * time.Millisecond)
defer c.Stop()
现在，当您想要进行服务器请求时，只需在实际请求之前插入
<- c.C
即可。
英文:
If you have a fixed upper limit on how many requests you can do in a particular amount of real time, you can use a time.NewTicker() to space things out.
c := time.NewTicker(50 * time.Millisecond)
defer c.Stop()
Now, when you want to make a server request, just insert
<- c.C
prior to the actual request.
答案2
得分: 0
我认为你可以尝试使用这个：https://github.com/uber-go/ratelimit
根据文档，它是并发安全的。
import (
"fmt"
"time"
"go.uber.org/ratelimit"
)
func main() {
rl := ratelimit.New(100) // 每秒限制100个请求
prev := time.Now()
for i := 0; i < 10; i++ {
now := rl.Take()
fmt.Println(i, now.Sub(prev))
prev = now
}
// 输出:
// 0 0
// 1 10ms
// 2 10ms
// 3 10ms
// 4 10ms
// 5 10ms
// 6 10ms
// 7 10ms
// 8 10ms
// 9 10ms
}
英文:
i think you can try this: https://github.com/uber-go/ratelimit
According to the documentation, it is concurrency safe.
import (
"fmt"
"time"
"go.uber.org/ratelimit"
)
func main() {
rl := ratelimit.New(100) // per second
prev := time.Now()
for i := 0; i < 10; i++ {
now := rl.Take()
fmt.Println(i, now.Sub(prev))
prev = now
}
// Output:
// 0 0
// 1 10ms
// 2 10ms
// 3 10ms
// 4 10ms
// 5 10ms
// 6 10ms
// 7 10ms
// 8 10ms
// 9 10ms
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论