Best way to bind shell output to a struct in Go?

huangapple
go评论111阅读模式
英文:

Best way to bind shell output to a struct in Go?

问题

我有一些在某些硬件上运行的芯片，并且我想将shell命令的输出绑定到一个结构体上，以便进行报告/日志记录。

  1. Num item1: 2
  2. INDEX LOAD MODEL_LOAD INST MEM  SHARE_MEM P2P_MEM DEVICE         NAMESPACE
  3. 1     2    3          4    50    600         700       1     1
  4. 1a     2b    3c          4c    5d    6e         7f       2     2
  5. Num item2: 2
  6. INDEX LOAD MODEL_LOAD INST MEM  SHARE_MEM P2P_MEM DEVICE         NAMESPACE
  7. 2a     2b    2c          3    0    0         0       1     1
  8. 1     0    0          0    0    0         0       2     2
  9. **************************************************

尝试

  1. cat out.txt | grep -i "Num $1" -A 3 | grep -i nvme | tr -s ' ' | cut -d' ' -f1-7

这实际上还不错，我可以传入一个参数，比如decoders或encoders，并获取每个芯片的负载指标。然而，我现在很好奇如何将其绑定到Go中的结构体上的最佳方法。

目前，我可以编写一个自定义的反序列化器，类似于：

  1. func main() {
  2. 	out, err := exec.Command("/bin/sh", "metrics.sh", "encoders").Output()
  3. 	if err != nil {
  4. 		fmt.Println(err)
  5. 		log.Fatal(err)
  6. 	}
  8. 	fmt.Println(string(out))
  9. }

但我觉得肯定有更好的方法，比如将输出转换为JSON并绑定到一个结构体之类的。

英文:

I have some chips running on some hardware and I want to bind the output of a shell command to a struct to do reporting/logging on.

  1. Num item1: 2
  2. INDEX LOAD MODEL_LOAD INST MEM  SHARE_MEM P2P_MEM DEVICE         NAMESPACE
  3. 1     2    3          4    50    600         700       1     1
  4. 1a     2b    3c          4c    5d    6e         7f       2     2
  5. Num item2: 2
  6. INDEX LOAD MODEL_LOAD INST MEM  SHARE_MEM P2P_MEM DEVICE         NAMESPACE
  7. 2a     2b    2c          3    0    0         0       1     1
  8. 1     0    0          0    0    0         0       2     2
  9. **************************************************

Attempt

  1. cat out.txt | grep -i "Num $1" -A 3 | grep -i nvme | tr -s ' ' | cut -d' ' -f1-7

This actually isn't too bad, I can pass in an arg like decoders or encoders and get the load metrics for each chip. However, I'm curious now the best way to bind this to a struct in Go.

Currently, what I can do is code up a custom deserializer from something like:

  1. func main() {
  2. 	out, err := exec.Command("/bin/sh", "metrics.sh", "encoders").Output()
  3. 	if err != nil {
  4. 		fmt.Println(err)
  5. 		log.Fatal(err)
  6. 	}
  8. 	fmt.Println(string(out))
  9. }

But I feel like there has to be a better way, like outputting as JSON and binding to a struct or something.

答案1

得分: 2

我会将您的输入文件转换为CSV格式，因为它适用于原始的表格数据，而且Go语言的标准库中有一个CSV编码器/解码器：

  1. awk -v OFS=',' '
  2.     $1 == "Num" {
  3.         count = $3
  4.         type = $2
  6.         getline
  7.         if ( !header++ ) {
  8.             $(NF+1) = "ID"
  9.             print
  10.         }
  11.         for ( id = 1; id <= count; id++ ) {
  12.             getline
  13.             $(NF+1) = type id
  14.             print
  15.         }
  16.     }
  17. ' file.txt

<sup>**警告：**该代码没有对字段进行CSV转义</sup>

  1. INDEX,LOAD,MODEL_LOAD,INST,MEM,SHARE_MEM,P2P_MEM,DEVICE,NAMESPACE,ID
  2. 1,2,3,4,50,600,700,/dev/nvme0,/dev/nvme0n1,decoders:1
  3. 1a,2b,3c,4c,5d,6e,7f,/dev/nvme1,/dev/nvme1n1,decoders:2
  4. 2a,2b,2c,3,0,0,0,/dev/nvme0,/dev/nvme0n1,encoders:1
  5. 1,0,0,0,0,0,0,/dev/nvme1,/dev/nvme1n1,encoders:2
  6. 0,0,0,0,0,0,0,/dev/nvme0,/dev/nvme0n1,scalers:1
  7. 1,0,0,0,0,0,0,/dev/nvme1,/dev/nvme1n1,scalers:2

**注意：**使用Go编写解析器来处理您的输入格式应该不难。

英文:

I would convert your input file to CSV, because it suits the original tabular data and also because the Go language have a CSV encoder/decoder in its standard library:

  1. awk -v OFS=&#39;,&#39; &#39;
  2.     $1 == &quot;Num&quot; {
  3.         count = $3
  4.         type = $2
  6.         getline
  7.         if ( !header++ ) {
  8.             $(NF+1) = &quot;ID&quot;
  9.             print
  10.         }
  11.         for ( id = 1; id &lt;= count; id++ ) {
  12.             getline
  13.             $(NF+1) = type id
  14.             print
  15.         }
  16.     }
  17. &#39; file.txt

<sup>Warning: the code doesn't CSV-escape the fields</sup>

  1. INDEX,LOAD,MODEL_LOAD,INST,MEM,SHARE_MEM,P2P_MEM,DEVICE,NAMESPACE,ID
  2. 1,2,3,4,50,600,700,/dev/nvme0,/dev/nvme0n1,decoders:1
  3. 1a,2b,3c,4c,5d,6e,7f,/dev/nvme1,/dev/nvme1n1,decoders:2
  4. 2a,2b,2c,3,0,0,0,/dev/nvme0,/dev/nvme0n1,encoders:1
  5. 1,0,0,0,0,0,0,/dev/nvme1,/dev/nvme1n1,encoders:2
  6. 0,0,0,0,0,0,0,/dev/nvme0,/dev/nvme0n1,scalers:1
  7. 1,0,0,0,0,0,0,/dev/nvme1,/dev/nvme1n1,scalers:2

N.B. Writing a parser in Go for your input format shouldn't be that difficult

答案2

得分: 2

如何直接使用Go语言处理你关心的文本呢？在Go语言中，你将比使用shell工具拥有更多的控制权。

下面是一个小型状态机，它会查找以"Num"开头的文本，以指示一个新的项目的开始。下一行是标题，会被跳过，接下来的行会被转换为一个行对象，并添加到该项目中。在项目之间的边界以及输入文本/文件的末尾，最后一个项目会被添加到所有项目的集合中。

  1. package main
  3. import (
  4. 	"bufio"
  5. 	"fmt"
  6. 	"regexp"
  7. 	"strings"
  8. )
  10. var txt = `
  11. Num item1: 2
  12. INDEX LOAD MODEL_LOAD INST MEM  SHARE_MEM P2P_MEM DEVICE         NAMESPACE
  13. 1     2    3          4    50    600         700       1     1
  14. 1a     2b    3c          4c    5d    6e         7f       2     2
  15. Num item2: 2
  16. INDEX LOAD MODEL_LOAD INST MEM  SHARE_MEM P2P_MEM DEVICE         NAMESPACE
  17. 2a     2b    2c          3    0    0         0       1     1
  18. 1     0    0          0    0    0         0       2     2
  19. Num item3: 1
  20. INDEX LOAD MODEL_LOAD INST MEM  SHARE_MEM P2P_MEM DEVICE         NAMESPACE
  21. i     iib    iic          iii    zero    zero         zero       i     i
  22. **************************************************
  23. `
  25. var columns = regexp.MustCompile(`\s+`)
  27. type Row struct {
  28. 	Index,
  29. 	Load,
  30. 	Model_Load,
  31. 	Inst_Mem,
  32. 	Share_Mem,
  33. 	P2p_Mem,
  34. 	Device,
  35. 	Namespace string
  36. }
  38. type Item []Row
  40. func main() {
  41. 	r := strings.NewReader(txt)
  42. 	scanner := bufio.NewScanner(r)
  44. 	items := make([]Item, 0)
  46. 	var item Item
  47. 	for scanner.Scan() {
  48. 		line := scanner.Text()
  49. 		line = strings.TrimSpace(line)
  51. 		if len(line) == 0 ||
  52. 			strings.HasPrefix(line, "***") {
  53. 			continue
  54. 		}
  56. 		// find beginning of an "item": if any previous item, save it and
  57. 		// reset item to append future rows; skip header line; continue
  58. 		if strings.HasPrefix(line, "Num item") {
  59. 			if len(item) > 0 {
  60. 				items = append(items, item)
  61. 				item = make(Item, 0)
  62. 			}
  63. 			scanner.Scan() // skip header
  64. 			continue
  65. 		}
  67. 		cols := columns.Split(line, -1)
  68. 		row := Row{cols[0], cols[1], cols[2], cols[3], cols[4], cols[5], cols[6], cols[7]}
  69. 		item = append(item, row)
  70. 	}
  72. 	// deal with last/trailing item
  73. 	if len(item) > 0 {
  74. 		items = append(items, item)
  75. 	}
  77. 	for i, item := range items {
  78. 		fmt.Printf("Item %d\n", i+1)
  79. 		for _, row := range item {
  80. 			fmt.Println(row)
  81. 		}
  82. 	}
  83. }

运行以上代码会输出以下结果：

  1. Item 1
  2. {1 2 3 4 50 600 700 1}
  3. {1a 2b 3c 4c 5d 6e 7f 2}
  4. Item 2
  5. {2a 2b 2c 3 0 0 0 1}
  6. {1 0 0 0 0 0 0 2}
  7. Item 3
  8. {i iib iic iii zero zero zero i}

我不知道有更好的方法来创建这个结构体，但它是直接的，而且相当清晰。

英文:

How about starting directly with the text you care about in Go? You'll have far more control in Go than you'll ever have with shell utilities.

This is a little state machine that looks for the leading text, "Num", to indicate the beginning of a new Item. The next line is the header, which is skipped, and following lines are converted to a Row, which are added to that Item. At the boundary between items and at the end of the input text/file, the last Item is added to the set of all Items.

  1. package main
  3. import (
  4. 	&quot;bufio&quot;
  5. 	&quot;fmt&quot;
  6. 	&quot;regexp&quot;
  7. 	&quot;strings&quot;
  8. )
  10. var txt = `
  11. Num item1: 2
  12. INDEX LOAD MODEL_LOAD INST MEM  SHARE_MEM P2P_MEM DEVICE         NAMESPACE
  13. 1     2    3          4    50    600         700       1     1
  14. 1a     2b    3c          4c    5d    6e         7f       2     2
  15. Num item2: 2
  16. INDEX LOAD MODEL_LOAD INST MEM  SHARE_MEM P2P_MEM DEVICE         NAMESPACE
  17. 2a     2b    2c          3    0    0         0       1     1
  18. 1     0    0          0    0    0         0       2     2
  19. Num item3: 1
  20. INDEX LOAD MODEL_LOAD INST MEM  SHARE_MEM P2P_MEM DEVICE         NAMESPACE
  21. i     iib    iic          iii    zero    zero         zero       i     i
  22. **************************************************
  23. `
  25. var columns = regexp.MustCompile(`\s+`)
  27. type Row struct {
  28. 	Index,
  29. 	Load,
  30. 	Model_Load,
  31. 	Inst_Mem,
  32. 	Share_Mem,
  33. 	P2p_Mem,
  34. 	Device,
  35. 	Namespace string
  36. }
  38. type Item []Row
  40. func main() {
  41. 	r := strings.NewReader(txt)
  42. 	scanner := bufio.NewScanner(r)
  44. 	items := make([]Item, 0)
  46. 	var item Item
  47. 	for scanner.Scan() {
  48. 		line := scanner.Text()
  49. 		line = strings.TrimSpace(line)
  51. 		if len(line) == 0 ||
  52. 			strings.HasPrefix(line, &quot;***&quot;) {
  53. 			continue
  54. 		}
  56. 		// find beginning of an &quot;item&quot;: if any previous item, save it and
  57. 		// reset item to append future rows; skip header line; continue
  58. 		if strings.HasPrefix(line, &quot;Num item&quot;) {
  59. 			if len(item) &gt; 0 {
  60. 				items = append(items, item)
  61. 				item = make(Item, 0)
  62. 			}
  63. 			scanner.Scan() // skip header
  64. 			continue
  65. 		}
  67. 		cols := columns.Split(line, -1)
  68. 		row := Row{cols[0], cols[1], cols[2], cols[3], cols[4], cols[5], cols[6], cols[7]}
  69. 		item = append(item, row)
  70. 	}
  72. 	// deal with last/trailing item
  73. 	if len(item) &gt; 0 {
  74. 		items = append(items, item)
  75. 	}
  77. 	for i, item := range items {
  78. 		fmt.Printf(&quot;Item %d\n&quot;, i+1)
  79. 		for _, row := range item {
  80. 			fmt.Println(row)
  81. 		}
  82. 	}
  83. }

That prints the following:

  1. Item 1
  2. {1 2 3 4 50 600 700 1}
  3. {1a 2b 3c 4c 5d 6e 7f 2}
  4. Item 2
  5. {2a 2b 2c 3 0 0 0 1}
  6. {1 0 0 0 0 0 0 2}
  7. Item 3
  8. {i iib iic iii zero zero zero i}

I don't know of a better way to create the struct, but it is direct, and fairly clean.

huangapple
  • 本文由 发表于 2022年9月21日 00:40:13
  • 转载请务必保留本文链接：https://go.coder-hub.com/73790001.html
  • awk
  • bash
  • go
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定