如何将具有可变结构的消息扁平化为protobuf?

huangapple go评论158阅读模式
英文:

How to flatten message with mutable structure into protobuf?

问题

我的proto buf格式如下:

package main;

message Test {
  optional string id = 1;
  optional string name = 2;
  optional string age = 3;
}

然后,我使用以下代码在golang中从输入中填充protobuf文件。str已经被解析。

test = &Test{
  id:   proto.String(str[0]),
  name: proto.String(str[1]),
  age:  proto.String(str[2]),
}

我需要处理的一个条件是Test结构中的一个或多个可选字段可能随机缺失,而我事先不知道。在golang中如何处理这种情况?

为了提供更多背景信息,文件中的真实数据可能如下所示:

id=1, name=peter, age=24
id=2, age=30
name=mary, age=31
id=100
name=bob
age=11

请注意,这只是一个翻译,我不会回答关于如何处理该问题的问题。

英文:

My proto buf format is this:

package main;

message Test {
  optional string id = 1;
  optional string name = 2;
  optional string age = 3;
}

Then I populate the protobuf files from the input in golang using the following code. str is already parsed.

test = &Test{
  id: proto.String(str[0]),
  name:  proto.String(str[1]),
  age: proto.String(str[2]),
},

One condition I have to handle is that one or more optional fields in the Test structure could be absent randomly and I do not know that in advance. How do I handle that in golang?

To give more context, the real data can look like these in the file:

id=1, name=peter, age=24
id=2, age=30
name=mary, age=31
id=100
name=bob
age=11

答案1

得分: 1

你可以使用正则表达式将输入字符串转换为有效的JSON,然后使用encoding/json包进行解析。这样做的好处是让json解析器为你处理一切。以下是你特定情况下的正则表达式。

基本上,该正则表达式查找field=value并替换为"field" : "value",然后用{}包裹起来创建有效的JSON。逗号保持不变。

这是你的代码示例:

package main

import (
	"encoding/json"
	"errors"
	"fmt"
	"log"
	"regexp"
)

var ins = []string{
	`id=1, name=peter, age=24`,
	`id=2, age=30`,
	`name=mary, age=31`,
	`id=100`,
	`name=bob`,
	`age=11`,
}

var ParseError = errors.New("bad parser input")
var Regex *regexp.Regexp

type Test struct {
	ID   string
	Name string
	Age  string
}

func (t *Test) String() string {
	return fmt.Sprintf("ID: %s, Name: %s, Age: %s", t.ID, t.Name, t.Age)
}

func main() {
	var err error
	Regex, err = regexp.Compile(`([^,\s]*)=([^,\s]*)`)
	if err != nil {
		log.Panic(err)
	}
	for _, v := range ins {

		test, err := ParseLine(v)
		if err != nil {
			log.Panic(err)
		}
		fmt.Println(test)
	}
}

func ParseLine(inp string) (*Test, error) {
	JSON := fmt.Sprintf("{%s}", Regex.ReplaceAllString(inp, `"$1" : "$2"`))
	test := &Test{}
	err := json.Unmarshal([]byte(JSON), test)
	if err != nil {
		return nil, err
	}
	return test, nil
}

以下是我认为是你所需的最小工作示例,但我对协议缓冲区不够熟悉,无法正确打印字符串...甚至无法验证它们是否正确。请注意,此示例无法在playground中运行。

package main

import (
	"errors"
	"fmt"
	"log"
	"regexp"

	"github.com/golang/protobuf/jsonpb"
	_ "github.com/golang/protobuf/proto"
)

var ins = []string{
	`id=1, name=peter, age=24`,
	`id=2, age=30`,
	`name=mary, age=31`,
	`id=100`,
	`name=bob`,
	`age=11`,
}

var ParseError = errors.New("bad parser input")
var Regex *regexp.Regexp

type Test struct {
	ID   *string `protobuf:"bytes,1,opt,name=id,json=id" json:"id,omitempty"`
	Name *string `protobuf:"bytes,2,opt,name=name,json=name" json:"name,omitempty"`
	Age  *string `protobuf:"bytes,3,opt,name=age,json=age" json:"age,omitempty"`
}

func (t *Test) Reset() {
	*t = Test{}
}

func (*Test) ProtoMessage() {}
func (*Test) Descriptor() ([]byte, []int) { return []byte{}, []int{0} }

func (t *Test) String() string {
	return fmt.Sprintf("ID: %v, Name: %v, Age: %v", t.ID, t.Name, t.Age)
}

func main() {
	var err error
	Regex, err = regexp.Compile(`([^,\s]*)=([^,\s]*)`)
	if err != nil {
		log.Panic(err)
	}
	for _, v := range ins {

		test, err := ParseLine(v)
		if err != nil {
			fmt.Println(err)
			log.Panic(err)
		}
		fmt.Println(test)
	}
}

func ParseLine(inp string) (*Test, error) {
	JSON := fmt.Sprintf("{%s}", Regex.ReplaceAllString(inp, `"$1" : "$2"`))
	test := &Test{}
	err := jsonpb.UnmarshalString(JSON, test)
	if err != nil {
		return nil, err
	}
	return test, nil
}

希望对你有所帮助!

英文:

You could use a regular expression to change your input strings into valid JSON, the use the encoding/json package to parse it. This has the advantage of letting the json parser take care of everything for you. Here is the regex for your particular case.

Basically, the regex looks for field=value and replaces with "field" : "value" and wraps it in {} to create valid JSON. The commas are left as is.

https://play.golang.org/p/_EEdTB6sve

package main
import (
"encoding/json"
"errors"
"fmt"
"log"
"regexp"
)
var ins = []string{
`id=1, name=peter, age=24`,
`id=2, age=30`,
`name=mary, age=31`,
`id=100`,
`name=bob`,
`age=11`,
}
var ParseError = errors.New("bad parser input")
var Regex *regexp.Regexp
type Test struct {
ID   string
Name string
Age  string
}
func (t *Test) String() string {
return fmt.Sprintf("ID: %s, Name: %s, Age: %s", t.ID, t.Name, t.Age)
}
func main() {
var err error
Regex, err = regexp.Compile(`([^,\s]*)=([^,\s]*)`)
if err != nil {
log.Panic(err)
}
for _, v := range ins {
test, err := ParseLine(v)
if err != nil {
log.Panic(err)
}
fmt.Println(test)
}
}
func ParseLine(inp string) (*Test, error) {
JSON := fmt.Sprintf("{%s}", Regex.ReplaceAllString(inp, `"$1" : "$2"`))
test := &Test{}
err := json.Unmarshal([]byte(JSON), test)
if err != nil {
return nil, err
}
return test, nil
}

Here is what I believe to be a minimum working case for what you are after, though I am not familiar enough with protocol buffers to get the strings to print right... or even verify if they are correct. Note that this doesn't run in the playground.

package main
import (
"errors"
"fmt"
"log"
"regexp"
"github.com/golang/protobuf/jsonpb"
_ "github.com/golang/protobuf/proto"
)
var ins = []string{
`id=1, name=peter, age=24`,
`id=2, age=30`,
`name=mary, age=31`,
`id=100`,
`name=bob`,
`age=11`,
}
var ParseError = errors.New("bad parser input")
var Regex *regexp.Regexp
type Test struct {
ID   *string `protobuf:"bytes,1,opt,name=id,json=id" json:"id,omitempty"`
Name *string `protobuf:"bytes,2,opt,name=name,json=name" json:"name,omitempty"`
Age  *string `protobuf:"bytes,3,opt,name=age,json=age" json:"age,omitempty"`
}
func (t *Test) Reset() {
*t = Test{}
}
func (*Test) ProtoMessage() {}
func (*Test) Descriptor() ([]byte, []int) {return []byte{}, []int{0}}
func (t *Test) String() string {
return fmt.Sprintf("ID: %v, Name: %v, Age: %v", t.ID, t.Name, t.Age)
}
func main() {
var err error
Regex, err = regexp.Compile(`([^,\s]*)=([^,\s]*)`)
if err != nil {
log.Panic(err)
}
for _, v := range ins {
test, err := ParseLine(v)
if err != nil {
fmt.Println(err)
log.Panic(err)
}
fmt.Println(test)
}
}
func ParseLine(inp string) (*Test, error) {
JSON := fmt.Sprintf("{%s}", Regex.ReplaceAllString(inp, `"$1" : "$2"`))
test := &Test{}
err := jsonpb.UnmarshalString(JSON, test)
if err != nil {
return nil, err
}
return test, nil
}

答案2

得分: 0

看起来你可以像下面这样为输入的每一行编写解析器。

注意:我没有使用proto值创建结构体,因为作为外部包,它无法在playground中导入。

package main

import (
	"errors"
	"fmt"
	"strings"
)

var ins = []string{
	`id=1, name=peter, age=24`,
	`id=2, age=30`,
	`name=mary, age=31`,
	`id=100`,
	`name=bob`,
	`age=11`,
}

var ParseError = errors.New("bad parser input")

type Test struct {
	ID   string
	Name string
	Age  string
}

func (t *Test) String() string {
	return fmt.Sprintf("ID: %s, Name: %s, Age: %s", t.ID, t.Name, t.Age)
}

func main() {
	for _, v := range ins {
		t, err := ParseLine(v)
		if err != nil {
			fmt.Println(err)
		} else {
			fmt.Println(t)
		}
	}
}

func ParseLine(inp string) (*Test, error) {
	splt := strings.Split(inp, ",")
	test := &Test{}
	for _, f := range splt {
		fieldVal := strings.Split(strings.TrimSpace(f), "=")
		switch strings.TrimSpace(fieldVal[0]) {
		case "id":
			test.ID = strings.TrimSpace(fieldVal[1])
		case "name":
			test.Name = strings.TrimSpace(fieldVal[1])
		case "age":
			test.Age = strings.TrimSpace(fieldVal[1])
		default:
			return nil, ParseError
		}
	}
	return test, nil
}

请注意,这是一个用Go语言编写的示例代码,用于解析输入字符串并创建相应的结构体对象。它将输入字符串按逗号分隔,并根据键值对的形式解析每个字段。如果解析过程中遇到无效的输入,将返回ParseError错误。在main函数中,它遍历了ins切片中的每个输入字符串,并调用ParseLine函数进行解析。如果解析成功,将打印出相应的结构体对象,否则将打印出错误信息。

英文:

Looks like you can write the parser for each line of your input something like the following.

NOTE: I didn't make the struct with proto values because as an external package, it can't be imported in the playground.

https://play.golang.org/p/hLZvbiMMlZ

package main
import (
"errors"
"fmt"
"strings"
)
var ins = []string{
`id=1, name=peter, age=24`,
`id=2, age=30`,
`name=mary, age=31`,
`id=100`,
`name=bob`,
`age=11`,
}
var ParseError = errors.New("bad parser input")
type Test struct {
ID   string
Name string
Age  string
}
func (t *Test) String() string {
return fmt.Sprintf("ID: %s, Name: %s, Age: %s", t.ID, t.Name, t.Age)
}
func main() {
for _, v := range ins {
t, err := ParseLine(v)
if err != nil {
fmt.Println(err)
} else {
fmt.Println(t)
}
}
}
func ParseLine(inp string) (*Test, error) {
splt := strings.Split(inp, ",")
test := &Test{}
for _, f := range splt {
fieldVal := strings.Split(strings.TrimSpace(f), "=")
switch strings.TrimSpace(fieldVal[0]) {
case "id":
test.ID = strings.TrimSpace(fieldVal[1])
case "name":
test.Name = strings.TrimSpace(fieldVal[1])
case "age":
test.Age = strings.TrimSpace(fieldVal[1])
default:
return nil, ParseError
}
}
return test, nil
}

huangapple
  • 本文由 发表于 2017年4月26日 02:08:19
  • 转载请务必保留本文链接:https://go.coder-hub.com/43618017.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定