你可以使用纯粹的bash和redis-cli来实现一个Redis作业队列吗?

huangapple go评论80阅读模式
英文:

Can I implement a redis job queue with plain bash and redis-cli?

问题

我对Redis相对较新,希望将作业队列实现为一个简单的作业ID列表(Redis中除此之外没有其他内容)。

我搜索了一下,但找不到如何在纯粹的Shell界面(redis-cli)中执行此操作的示例。示例主要使用某些编程语言,如NodeJS、GOJavaPython。但我不想受限于其中的任何一种语言。为了减少依赖性,我希望在纯bash/shell中实现工作队列,因为我的应用需求不是很复杂。

具体而言,我有一个预定义的作业列表,每个作业都有一个整数ID。我打算设置一个Redis服务器和多个工作节点,每个节点都会从Redis队列/列表中依次获取/删除作业ID(然后处理它)。

根据阅读的相关文章,我猜想以下步骤:

  1. 通过RPUSH将所有作业ID初始化到Redis服务器中的列表中。

  2. 每个节点以某种方式从列表中获取并删除一个作业ID,计算它,然后将少量结果(以JSON编码值的形式)保存到Redis中的另一个数据结构中。

我的问题是:

是否可以使用纯bash脚本和redis-cli来完成这个任务,还是需要一个完整的编程语言?

在第2步中,节点之间的redis-cli实例如何处理队列的并发使用?也就是说,如果一个节点正在删除一个作业ID,确保集群中的其他节点不会再次读取/看到相同的ID。哪些命令适用于此?

顺便说一下,这都是在Ubuntu 22.04上进行的。没有并行处理或过期作业,每个节点按顺序获取一个作业,完成它,然后获取下一个,直到没有更多的作业要获取。另外,我将使用单个Redis服务器,而不是Redis服务器集群,以保持简单。

英文:

I am relatively new to redis, and I am looking to implement a job queue as a simple list of job ids (and nothing else in redis).

I've searched around, but couldn't find examples of how to do this in plain shell interface (redis-cli). The examples are mostly using some programming languages such as NodeJS, GO, Java, and Python. But I don't want to be bound to one of these languages. To reduce dependency, I would like to implement the workqueue in plain bash/shell, since my application need is not that complex.

Specifically, I have a pre-defined list of jobs, each with an integer id. I intend to set up a redis-server and multiple worker nodes that each takes/deletes job ids one at a time from the redis queue/list (and then processes it).

Based on reading the linked articles, I guess the steps are:

  1. initialize the list by RPUSH-ing all job ids to the redis-server.

  2. each node somehow takes and deletes a job id from the list, compute it, and then save a small amount of result (as a json encoded value) to another data structure in redis.

My questions are:

Can this be done using plain bash scripts and redis-cli, or is there a reason why a full-fledged programming language is required?

How can redis-cli instances among the nodes handle concurrent use of the queue in Point 2? That is, if one node is deleting a job id, make sure the other nodes in the cluster do not read/see the same id again. What commands are appropriate for this?

By the way, this is all with Ubuntu 22.04. There is no parallelism or expiration, and each node sequentially takes one job, completes it, and takes next, until there is no more jobs to take. Also, I'd use a single redis-server and not a cluster of redis-servers, for simplicity.

答案1

得分: 0

这应该足够简单。Redis 是单线程的,所有命令都是原子的。

要将内容推送到列表,请使用 LPUSH 命令:

$ redis-cli LPUSH foo 1
(integer) 1

要从列表中弹出内容,请使用 RPOP 命令:

$ redis-cli RPOP foo
1) "foo"
2) "1"

如果您希望在列表为空时进行等待,只需使用 BRPOP 命令:

$ redis-cli BRPOP foo 60
1) "foo"
2) "1"

由于每个命令都是原子的,只有一个 RPOP 或 BRPOP 命令调用将接收到从列表中弹出的任何给定项。

您可以很容易地通过在几个终端窗口中尝试几次 BRPOP 调用,然后使用 LPUSH 添加内容到列表来测试这一点。只有一个终端会获得弹出的值。

英文:

This should be simple enough. Redis is single-threaded and all commands are atomic.

To push things to the List use LPUSH:

$ redis-cli LPUSH foo 1
(integer) 1

To pop things from the List use RPOP:

$ redis-cli RPOP foo
1) "foo"
2) "1"

If you want to wait for something if the List is empty, just use BRPOP instead:

$ redis-cli BRPOP foo 60
1) "foo"
2) "1"

Since each command is atomic, only a single call to RPOP or BRPOP will receive any given item popped to the List.

You can test this out easily enough by trying a couple of BRPOP calls in a couple terminal windows and then adding something to the List using LPUSH. Only one terminal will get the popped value.

huangapple
  • 本文由 发表于 2023年6月26日 05:31:38
  • 转载请务必保留本文链接:https://go.coder-hub.com/76552493.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定