`exec.Command()`在运行`kcat`时没有输出。

huangapple go评论80阅读模式
英文:

`exec.Command()` produces no output when running `kcat`

问题

我需要将kcat封装在一个Go函数中以读取一系列主题消息,所以想使用exec.Command()来实现,代码如下:

package main

import (
	"fmt"
	"os/exec"
)

func main() {

	cmd := exec.Command("kcat", "-b kafka.kafka.svc.cluster.local:9092", "-t messages", "-o 11000", "-c 11333")

	fmt.Println("Command String:", cmd.String())

	out, err := cmd.CombinedOutput()

	if err != nil {
		fmt.Println("Error Accessing kafka topic messages ", err.Error(), string(out))
		return
	}

	fmt.Println("Result Length:", len(out))

	fmt.Println("Result Content:", string(out))

}

然而,这只返回了kcat输出的第一行:

/app/tools # ./five
Command String: /usr/bin/kcat -b kafka.kafka.svc.cluster.local:9092 -t messages -o 11000 -c 11333
Result Length: 58

Result Content: % Auto-selecting Producer mode (use -P or -C to override)

(注意:我在Docker容器中运行这个命令,但我认为这没有影响)

然而,当直接从CLI运行时,它可以正常工作:

/app/tools # 
/app/tools # kcat -b kafka.kafka.svc.cluster.local:9092 -t messages -o 10 -c 15
% Auto-selecting Consumer mode (use -P or -C to override)
%4|1640957136.462|OFFSET|rdkafka#consumer-1| [thrd:main]: messages [1]: offset reset (at offset 10) to END: fetch failed due to requested offset not available on the broker: Broker: Offset out of range
%4|1640957136.483|OFFSET|rdkafka#consumer-1| [thrd:main]: messages [2]: offset reset (at offset 10) to END: fetch failed due to requested offset not available on the broker: Broker: Offset out of range
[{"Name":"newOrder", "ID":"9266","Time":"9266","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"1547","Time":"1547","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"9179","Time":"9179","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"8740","Time":"8740","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"9318","Time":"9318","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"1743","Time":"1743","Data":"new order", "Eventname":"newOrder"}]

kcat命令似乎有一些独特之处,导致在Go中的exec.Command()无法正常工作。

问题:

  • 是否有其他方法可以在Go中实现相同的效果?
  • 这可能是我使用exec.Command()的问题吗?

理想情况下,我希望在这种情况下使用kcat命令,因为我想避免在这个实例中使用segmentios的kafka-go库。

[编辑]

  • 将参数分开(由@onecricketeer建议):
cmd := exec.Command("kcat", "-b", "kafka.kafka.svc.cluster.local:9092", "-t", "messages", "-o", "11000", "-c", "11333")

结果(相同的错误):

/app/tools # ./code
Command String: /usr/bin/kcat -b kafka.kafka.svc.cluster.local:9092 -t messages -o 11000 -c 11333
Result Length: 58
Result Content: % Auto-selecting Producer mode (use -P or -C to override)
  • 使用BASH作为shell(由maxm建议):

相同的结果,即只报告kcat输出的第一行:

/app/tools # ./code
Command String: /bin/bash -c kcat -b kafka.kafka.svc.cluster.local:9092 -t messages -o 11000 -c 11333
Result Length: 58
Result Content: % Auto-selecting Producer mode (use -P or -C to override)

[编辑]

注意:然而,当我使用Python的shell执行机制时,它可以正常工作,这让我想知道Go的shell处理功能是否有问题:

import subprocess

process = subprocess.Popen(["kcat","-b","kafka.kafka.svc.cluster.local:9092","-t","messages","-o","1", "-c", "11"], 
                           stdout=subprocess.PIPE,
                           universal_newlines=True)

while True:
    output = process.stdout.readline()
    print(output.strip())
    # Do something else
    return_code = process.poll()
    if return_code is not None:
        print('RETURN CODE', return_code)
        # Process has finished, read rest of the output 
        for output in process.stdout.readlines():
            print(output.strip())
        break

结果:

/app/tools/python # python3 code.py 
% Auto-selecting Consumer mode (use -P or -C to override)
%4|1641004616.232|OFFSET|rdkafka#consumer-1| [thrd:main]: messages [2]: offset reset (at offset 1) to END: fetch failed due to requested offset not available on the broker: Broker: Offset out of range
%4|1641004616.236|OFFSET|rdkafka#consumer-1| [thrd:main]: messages [1]: offset reset (at offset 1) to END: fetch failed due to requested offset not available on the broker: Broker: Offset out of range
[{"Name":"newOrder", "ID":"4512","Time":"4512","Data":"new order", "Eventname":"newOrder"}]

RETURN CODE 0
[{"Name":"newOrder", "ID":"2388","Time":"2388","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"8707","Time":"8707","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"1643","Time":"1643","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"2421","Time":"2421","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"7520","Time":"7520","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"1258","Time":"1258","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"1457","Time":"1457","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"2907","Time":"2907","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"9266","Time":"9266","Data":"new order", "Eventname":"newOrder"}]

[{"Name":"newOrder", "ID":"1547","Time":"1547","Data":"new order", "Eventname":"newOrder"}]

英文:

I need to wrap kcat in a Go function to read a series of topic messages, so thought to use exec.Command() for this is as follows:

package main
import (
"fmt"
"os/exec"
)
func main() {
cmd := exec.Command("kcat", "-b kafka.kafka.svc.cluster.local:9092", "-t messages", "-o 11000", "-c 11333")
fmt.Println("Command String:", cmd.String())
out, err := cmd.CombinedOutput()
if err != nil {
fmt.Println("Error Accessing kafka topic messages ", err.Error(), string(out))
return
}
fmt.Println("Result Length:", len(out))
fmt.Println("Result Content:", string(out))
}

However, this returns only the first line of the kcat output:

/app/tools # ./five
Command String: /usr/bin/kcat -b kafka.kafka.svc.cluster.local:9092 -t messages -o 11000 -c 11333
Result Length: 58

Result Content: % Auto-selecting Producer mode (use -P or -C to override)

(NOTE: I'm running this within a docker container, however I don't think it makes a difference)

However, this works fine when run directly from the CLI:

/app/tools # 
/app/tools # kcat -b kafka.kafka.svc.cluster.local:9092 -t messages -o 10 -c 15
% Auto-selecting Consumer mode (use -P or -C to override)
%4|1640957136.462|OFFSET|rdkafka#consumer-1| [thrd:main]: messages [1]: offset reset (at offset 10) to END: fetch failed due to requested offset not available on the broker: Broker: Offset out of range
%4|1640957136.483|OFFSET|rdkafka#consumer-1| [thrd:main]: messages [2]: offset reset (at offset 10) to END: fetch failed due to requested offset not available on the broker: Broker: Offset out of range
[{"Name":"newOrder", "ID":"9266","Time":"9266","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"1547","Time":"1547","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"9179","Time":"9179","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"8740","Time":"8740","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"9318","Time":"9318","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"1743","Time":"1743","Data":"new order", "Eventname":"newOrder"}]

There seems to be something unique about the kcat command which breaks exec.Command() in Go.

Questions:

  • Is there any other way I could achieve the same effect in Go?
  • Is this perhaps an issue with the way I'm using exec.Command()

Ideally, I can use the the kcat command in this case as I'd like to avoid using segmentios kafka-go library in this instance.

[EDIT]

  • Separating the arguments (as suggested by @onecricketeer):
cmd := exec.Command("kcat", "-b", "kafka.kafka.svc.cluster.local:9092", "-t", "messages", "-o", "11000", "-c", "11333")

Result (same error):

/app/tools # ./code
Command String: /usr/bin/kcat -b kafka.kafka.svc.cluster.local:9092 -t messages -o 11000 -c 11333
Result Length: 58
Result Content: % Auto-selecting Producer mode (use -P or -C to override)
  • Using BASH as the shell (suggested by maxm):

Same result, i.e only the first line of the kcat output is reported:

/app/tools # ./code
Command String: /bin/bash -c kcat -b kafka.kafka.svc.cluster.local:9092 -t messages -o 11000 -c 11333
Result Length: 58
Result Content: % Auto-selecting Producer mode (use -P or -C to override)

[EDIT]

NOTE: However, when I use Python's shell execution mechanism, it works fine, which leads me to wonder if there is something defective about Gos shell handling features:

import subprocess
process = subprocess.Popen(["kcat","-b","kafka.kafka.svc.cluster.local:9092","-t","messages","-o","1", "-c", "11"], 
stdout=subprocess.PIPE,
universal_newlines=True)
while True:
output = process.stdout.readline()
print(output.strip())
# Do something else
return_code = process.poll()
if return_code is not None:
print('RETURN CODE', return_code)
# Process has finished, read rest of the output 
for output in process.stdout.readlines():
print(output.strip())
break

Results:

/app/tools/python # python3 code.py 
% Auto-selecting Consumer mode (use -P or -C to override)
%4|1641004616.232|OFFSET|rdkafka#consumer-1| [thrd:main]: messages [2]: offset reset (at offset 1) to END: fetch failed due to requested offset not available on the broker: Broker: Offset out of range
%4|1641004616.236|OFFSET|rdkafka#consumer-1| [thrd:main]: messages [1]: offset reset (at offset 1) to END: fetch failed due to requested offset not available on the broker: Broker: Offset out of range
[{"Name":"newOrder", "ID":"4512","Time":"4512","Data":"new order", "Eventname":"newOrder"}]
RETURN CODE 0
[{"Name":"newOrder", "ID":"2388","Time":"2388","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"8707","Time":"8707","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"1643","Time":"1643","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"2421","Time":"2421","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"7520","Time":"7520","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"1258","Time":"1258","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"1457","Time":"1457","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"2907","Time":"2907","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"9266","Time":"9266","Data":"new order", "Eventname":"newOrder"}]
[{"Name":"newOrder", "ID":"1547","Time":"1547","Data":"new order", "Eventname":"newOrder"}]

答案1

得分: 1

go命令:

cmd := exec.Command("kcat", "-b kafka.kafka.svc.cluster.local:9092", "-t messages", "-o 11000", "-c 11333")

与shell命令相同:

kcat "-b kafka.kafka.svc.cluster.local:9092" "-t messages" "-o 11000" "-c 11333"

你需要分隔参数,就像shell默认在每个空格上为你做的那样:

cmd := exec.Command("kcat", "-b", "kafka.kafka.svc.cluster.local:9092", "-t", "messages", "-o", "11000", "-c", "11333")
英文:

The go command:

cmd := exec.Command("kcat", "-b kafka.kafka.svc.cluster.local:9092", "-t messages", "-o 11000", "-c 11333")

is the same as the shell command:

kcat "-b kafka.kafka.svc.cluster.local:9092" "-t messages" "-o 11000" "-c 11333"

You need to separate your arguments, the same as the shell is doing for you on every space by default:

cmd := exec.Command("kcat", "-b", "kafka.kafka.svc.cluster.local:9092", "-t", "messages", "-o", "11000", "-c", "11333")

答案2

得分: 1

根据输出,生产者模式正在自动选择。

尝试使用消费者模式,并使用分离的参数。

cmd := exec.Command("kcat", "-C", "-b", "kafka.kafka.svc.cluster.local:9092", "-t", "messages", "-o", "11000", "-c", "11333")
英文:

As the output says, producer mode is being auto selected

Try using consumer mode with separated arguments

cmd := exec.Command("kcat", "-C", "-b", "kafka.kafka.svc.cluster.local:9092", "-t", "messages", "-o", "11000", "-c", "11333")

huangapple
  • 本文由 发表于 2021年12月31日 22:30:27
  • 转载请务必保留本文链接:https://go.coder-hub.com/70543222.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定