英文:
Most efficient way to run a function on every time a timer finishes, but with a large number of timers (thousands/millions)
问题
我正在运行一个服务，用户上传一个持续时间，每次计时器结束时都必须重复执行一个函数。
例如，用户说“每5分钟运行一次”，然后这个函数必须每5分钟运行一次。这是通过一个API完成的。
对于少量的计时器，这很简单：
func doEvery(d time.Duration, f func(time.Time)) {
for x := range time.Tick(d) {
f(x) // 每隔d时间运行函数
}
}
我可以在每个计时器中运行一个goroutine，这样可以正常工作。我可以使用一些基本的WaitGroups和同步功能来启动和停止所有计时器。
但是如果我有成千上万甚至百万个计时器怎么办？我可以为每个计时器创建一个goroutine，但这感觉非常低效。这不是Erlang。
我应该有多个工作队列，按“延迟”排序，并为更频繁的函数分配更多的工作线程吗？如果计时器还没有准备好，那么它会被放回队列。
这也不是理想的解决方案，因为工作线程在忙等待（弹出，检查时间，推入队列）而不是阻塞直到下一个计时器完成。
也许我可以使用某种映射，以剩余持续时间为索引？我不确定这里最好的方法是什么。
英文:
I'm running a service where users upload a duration and a function must be repeatedly executed every time the timer runs out.
For example, the user says "Run every 5 minutes" and then this function must be run every 5 minutes. This is done through an API.
For a small numbers of timers, this is trivial:
func doEvery(d time.Duration, f func(time.Time)) {
for x := range time.Tick(d) {
f(x) // Run the function every d duration
}
}
I can run each timer in a goroutine, and that works fine. I can start and stop everything with some basic WaitGroups and sync functionality.
But what if I have thousands or millions of timers? I can create a goroutine for each one, but that feels very inefficient. This isn't Erlang.
Should I have multiple work queues, sorted by the "delay", and simply allocate more workers for the more frequent functions? If the timer isn't ready, then it's put back on the queue.
This also isn't ideal, because the workers are busy-waiting (popping off, checking times, pushing on the queue) rather than blocking until the next timer finishes.
Maybe I could have some sort of map, indexed by the remaining duration? I'm not sure what the best approach is here.
答案1
得分: 1
我最近构建了这样一种解决方案，用户可以根据他们的间隔设置收到通知。我使用RabbitMQ和两个Go脚本构建了一个工作池模型。其中一个Go脚本使用下面的cron模块在消息队列中创建作业：
https://github.com/robfig/cron
另一个Go脚本是消费消息队列并采取行动的工作脚本。
为了扩展，我运行多个工作脚本实例。可以通过RabbitMQ中的消息数量来自动扩展工作脚本的数量。
英文:
I recently built this kind of solution where the user gets notification based on their interval settings. I make a worker pool model using rabbitMQ and two go script. Where one go script uses below cron module to create jobs in a message queue
https://github.com/robfig/cron
Another go script is the worker script consuming message queue and taking action.
To scale, I'm running multiple instances of worker script. Autoscale of worker script can be done by the number of messages in rabbitMQ
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论