How to make a Go app wait for data to become available in a Redis list?

huangapple go评论94阅读模式
英文:

How to make a Go app wait for data to become available in a Redis list?

问题

我有一个使用Go语言的应用程序,它使用Go的Radix Redis客户端。它的任务是作为后台进程运行,并等待Redis列表中添加项目以进行处理。

背景(可以跳过):我有一个Node.js的Web应用程序,需要从外部API请求数据,已知该API的响应时间至少需要一秒钟或更长时间。数据不需要立即使用,所以我让Node应用程序将一个项目添加到Redis列表中,并继续处理重要的事情。同时,我有一个Go应用程序,应该在后台运行,并在发现列表中添加了项目时执行一些工作。

目前,这只是一个main函数,它连接到Redis,并使用Redis的BRPOP命令检查列表中是否有项目(因为我希望Redis在等待作业时阻塞)。运行程序会建立连接,然后...

  • 如果列表中有项目,Go应用程序会根据需要处理它
  • 如果列表中没有项目,Go应用程序就会退出

我希望Go应用程序只是等待新项目,而且我真的不想使用轮询,特别是因为BRPOPBLPUSH命令专门用于应用程序无需轮询。

我目前的解决方案是创建一个无限循环,不断运行BRPOP命令。但这感觉不是正确的方法。

以下是到目前为止的整个程序(我对Go语言完全不熟悉,但对编程不陌生,所以请不要笑得太厉害):

package main

import (
  "fmt"
  "github.com/fzzy/radix/redis"
)

func main() {
  client, err := redis.Dial("tcp", "localhost:6379")

  if err != nil {
    fmt.Println("An error message just for debug purposes")
  } else {
    // 这里我检查服务器是否响应
    ping := client.Cmd("PING")
    fmt.Println(ping) // 为了确保我做得对

    for {
      fmt.Println("Looking for a new Q item...")
      workItem, err := client.Cmd("BRPOP", "q:test").Str()
      
      if err != nil {
        fmt.Println("No error, " + workItem)
      }
    }
  }
}

那么,什么是使Go应用程序等待Redis列表中的数据变得可用的正确方法?

更新:根据一些评论,我将进一步解释我期望的和实际发生的情况。

我对阻塞式Redis命令的理解是,它会暂停调用它的程序,直到有东西返回。所以在我的情况下,当在队列中没有项目的情况下运行程序时,程序会无限打印:

Looking for a new Q item...
No error, 

它无限打印上述内容是有道理的,因为我有一个无限循环。但是,由于我使用的是阻塞命令,我希望它的行为更像是推送。我希望循环在队列中有项目之前暂停并等待,然后再进行下一次迭代。我担心即使没有队列项目,它仍然继续循环,因为不断发送BRPOP命令(尽管它是一个阻塞命令)会不必要地消耗资源并降低Redis的性能。

所以现在新的问题是:

  • 是否有更好的方法让Go程序在等待Redis列表(或我自己创建的队列)中的新项目时保持运行?
  • 在当前形式下运行此程序是否会消耗比应有的更多的资源,并且是否会降低Redis的性能?

解决方案

感谢@JimB关于“你忽略的错误...”的问题,我再次查看了代码,确保没有漏掉明显的问题。果然,我犯了个错误:原来在循环中我混淆了错误处理和成功处理的代码。所以每次出错时,我都会打印一个成功的消息到控制台,以便我能看到程序内部发生了什么。在更正代码并正确处理错误后,我发现我使用了错误的参数调用了BRPOP。我修复了代码,现在循环将一直停在那里,直到列表中有要处理的内容。

以下是更正后的代码:

// 现在我们还要导入"log"
// 在循环内部...
if err != nil {
  log.Fatal(err) // 这让我注意到了参数数量错误的问题
} else {
  fmt.Println("No error to be had " + workItem) // 只有在列表中有项目时才会打印
}

我想我一直都做得对。如果对性能有任何评论或建议,或者有更好的方法来做这个,我仍然愿意听取并希望能听到。

英文:

I have a Go app that is using the Radix Redis client for Go. Its job is to run as a background process and wait for a Redis list to have items added to it for processing.

Background (feel free to skip): I have a Node.js web app that has a need to request data from an external API that's been known to take at least an entire second or more to respond. The data isn't required immediately so I'm having the Node app add an item to a Redis list and move on to the important stuff. Meanwhile, I have a Go app that is supposed to run behind the scenes and do some work each time it finds an item has been added to the list.

Right now it's just a main function that connects to Redis and checks to see if a list has items in it using the Redis BRPOP command (because I want Redis to block while its waiting for jobs). Running the program results in a connection being made and then...

  • If there's an item in a list, the Go app processes it however it needs to
  • If there's no items in the list then the Go app just quits

I need the Go app to just sit there and wait for new items and I really don't want to resort to polling especially since the BRPOP and BLPUSH commands are made specifically so apps don't need to poll.

My current solution is to just create an infinite loop that runs the BRPOP command over and over again forever. This feels like the wrong way to do it.

Here's the entire program so far (I'm completely new to Go but not to programming so don't laugh too hard):

package main

import (
  "fmt"
  "github.com/fzzy/radix/redis"
)

func main() {
  client, err := redis.Dial("tcp", "localhost:6379")

  if err != nil {
    fmt.Println("An error message just for debug purposes")
  } else {
    // Here I check to make sure the server is responding
    ping := client.Cmd("PING")
    fmt.Println(ping) // Just to be sure I'm doing things right
    
    for {
      fmt.Println("Looking for a new Q item...")
      workItem, err := client.Cmd("BRPOP", "q:test").Str()
      
      if err != nil {
        fmt.Println("No error, " + workItem)
      }
    }
  }
}

So what is the proper way to make a Go app wait for data to become available in a Redis list?

Update: Based on some of the comments I'll explain a little further what I'm expecting and what's happening.

My understanding of a blocking Redis command is that it will just pause the program calling it until it has something to return. So in my case, when running the program with no items in the queue the program prints:

Looking for a new Q item...
No error, 

It prints the above infinitely which makes sense since I have a never-ending for loop but because I'm using the blocking command, I was expecting it to behave more like a push. I expected the loop to just pause and wait until there's an item in the queue before moving on to the next iteration. My concern with it continuing to loop even with no queue items is that eventually the process will start to consume resources unnecessarily and slow down Redis because of the constant barrage of BRPOP commands (despite it being a blocking command).

So I suppose now the new questions would be

  • Is there a better way to keep the Go program running while waiting for new items in the Redis list (or my makeshift queue)?
  • Will running this program in its current form consume more resources than it should and/or degrade Redis performance?

Solved

Thanks to @JimB's question about the "error you're ignoring..." I took a second look at the code to make sure I wasn't missing something obvious. I was: Turns out that in my loop I had switched up the error and success handling code. So each time it errored out I would print a success message to the console just so I could see what was happening inside the program. After switching the code and correctly handling the error I found that I was calling BRPOP with too few arguments. I fixed the code and now the loop will just hang there until there's something in the list to handle.

Corrected lines below:

// We now import "log" now as well
// Inside the for loop...
if err != nil {
  log.Fatal(err) // This alerted me to the wrong # of arguments issue
} else {
  fmt.Println("No error to be had " + workItem) // Will not print until there is an item in the list
}

I guess I was doing it right all along. If there are any comments or suggestions about performance or better ways to do this, however, I'm still all hears and would love to hear them.

答案1

得分: 2

@JimB应该得到赞誉。我改变了错误和成功处理程序的顺序。我给BRPOP命令提供的参数太少了。循环内的以下代码修复了这个问题:

// 现在我们也导入"log"
// 在for循环内...
if err != nil {
  log.Fatal(err) // 这让我意识到了参数数量错误的问题
} else {
  fmt.Println("没有错误 " + workItem) // 只有在列表中有项目时才会打印
}

谢谢大家。

英文:

@JimB deserves the credit. I switched up the error and success handlers. I was giving too few arguments to the BRPOP command. This code within the loop fixed it:

// We now import "log" now as well
// Inside the for loop...
if err != nil {
  log.Fatal(err) // This alerted me to the wrong # of arguments issue
} else {
  fmt.Println("No error to be had " + workItem) // Will not print until there is an item in the list
}

Thanks everyone.

huangapple
  • 本文由 发表于 2015年4月2日 23:33:08
  • 转载请务必保留本文链接:https://go.coder-hub.com/29416837.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定