Unmarshal XML with variable child types while preserving order of children

I have an XML structure with a variable set of child elements. I want to unmarshal these child objects into a structure while preserving their order. I am currently using encoding/xml to unmarshal the xml, but this is not a strict requirement.

Sample XML structure:

<protocol>
    // ... more packet elements
    <packet family="Npc" action="Player">
        <comment>Main NPC update message</comment>
        <array name="positions" type="NpcUpdatePosition"/>
        <break/>
        <array name="attacks" type="NpcUpdateAttack"/>
        <break/>
        <array name="chats" type="NpcUpdateChat"/>
        <break/>
        <field name="hp" type="short" optional="true"/>
        <field name="tp" type="short" optional="true"/>
    </packet>
    // ... more packet elements
</protocol>

The variable elements to which I'm referring are child elements of the packet elements.

My models look like this:

type Protocol struct {
	Packets []ProtocolPacket `xml:"packet"`
}

type ProtocolPacket struct {
	Family       string                `xml:"family,attr"`
	Action       string                `xml:"action,attr"`
	Instructions /* ??? */             `xml:",any"`
	Comment      string                `xml:"comment"`
}

In this XML spec, there are a number of different elements such as array, break, and field, shown in the above sample, that need to be coalesced into a single slice while maintaining their order. These are referred to generally as "instructions". (comment in the example is a special case that should only ever be seen once).

I'm completely stumped on how to model the list of "instructions". One idea I had was to create an interface ProtocolInstruction with a custom unmarshaller that assigned an implementation depending on the element type, but I don't think this pattern would work, as you need to know the receiver type ahead of time for the unmarshal function to satisfy the appropriate interface.

I came across this question, but the suggested answer does not preserve the order of elements between different element names. Another idea I had was to use this method but write custom unmarshallers for each type that increment a counter and store the element index - that way even if order isn't preserved, it can at least be retrieved. However, this seems like a lot of work and a messy implementation, so I'm searching for alternatives.

Is there any way to unmarshal variable child XML elements while preserving their order in go?

Solution 1

Drawing on the highest-rated answer (so far) to unmarshal extra attributes, you could create the simple structs:

type Protocol struct {
	Packets []Packet `xml:"packet"`
}

type Packet struct {
	Family  string `xml:"family,attr"`
	Action  string `xml:"action,attr"`
	Comment string `xml:"comment"`

	Instructions []Instruction `xml:",any"`
}

type Instruction struct {
	Name  xml.Name
	Attrs []xml.Attr `xml:",any,attr"`
}

Any elements in a packet not handled by the more precise rules at the top of the Packet struct will be passed to Instruction which will decode the element into its name and a slice of its attributes.

Unmarshalling your sample XML will produce a var protocol with Packets.Instructions that contain rather raw XML values (which you can see me handling in the String method, later):

var protocol Protocol
xml.Unmarshal([]byte(opXML), &protocol)

for _, it := range protocol.Packets[0].Instructions {
	fmt.Println(it)
}

{name:array attrs:{name:positions type:NpcUpdatePosition}}
{name:break attrs:{}}
{name:array attrs:{name:attacks type:NpcUpdateAttack}}
{name:break attrs:{}}
{name:array attrs:{name:chats type:NpcUpdateChat}}
{name:break attrs:{}}
{name:field attrs:{name:hp type:short optional:true}}
{name:field attrs:{name:tp type:short optional:true}}

The String method for Instruction:

func (it Instruction) String() (s string) {
	s += fmt.Sprintf("{name:%s", it.Name.Local)
	s += " attrs:{"
	sep := ""
	for _, attr := range it.Attrs {
		s += fmt.Sprintf("%s%s:%s", sep, attr.Name.Local, attr.Value)
		sep = " "
	}
	s += "}}"
	return
}

Solution 2

The accepted answer for the same question exmeplifies making your own unmarshaller, like you suggested. I don't know what kind of structure you expect. I don't know generics (maybe there's a cleaner solution with generics), so I came up with the following. The Protocol and Packet structs remain the same, the big change comes with Instruction:

type Instruction struct {
	name string

	arrayAttrs struct{ name, type_ string }
	fieldAttrs struct {
		name, type_ string
		optional    bool
	}
}

and its unmarshaller:

type Instruction struct {
	name string

	array *Array
	field *Field
}

type Array struct {
	name, type_ string
}

type Field struct {
	name, type_ string
	optional    bool
}

func (it *Instruction) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
	it.name = start.Name.Local

	switch it.name {
	case "array":
		it.array = &Array{}
		for _, attr := range start.Attr {
			value := attr.Value
			switch attr.Name.Local {
			case "name":
				it.array.name = value
			case "type":
				it.array.type_ = value
			}
		}
	case "field":
		it.field = &Field{}
		for _, attr := range start.Attr {
			value := attr.Value
			switch attr.Name.Local {
			case "name":
				it.field.name = value
			case "type":
				it.field.type_ = value
			case "optional":
				vb, _ := strconv.ParseBool(value)
				it.field.optional = vb
			}
		}
	}

	return d.Skip()
}

func (it Instruction) String() (s string) {
	switch it.name {
	case "array":
		s = fmt.Sprintf("{array: {name:%s type:%s}}", it.array.name, it.array.type_)
	case "break":
		s = "{break: {}}"
	case "field":
		s = fmt.Sprintf("{field: {name:%s type:%s optional:%t}}", it.field.name, it.field.type_, it.field.optional)
	}
	return
}

Using the same unmarshalling code in main (from above):

{array: {name:positions type:NpcUpdatePosition}}
{break: {}}
{array: {name:attacks type:NpcUpdateAttack}}
{break: {}}
{array: {name:chats type:NpcUpdateChat}}
{break: {}}
{field: {name:hp type:short optional:true}}
{field: {name:tp type:short optional:true}}

Solution 3

Drawing inspiration from the RawMessage (Unmarshal) example in the JSON documentation, it looks like embracing the any type can allow the simplest struct representation I've tried so far:

type Protocol struct {
	Packets []Packet `xml:"packet"`
}

type Packet struct {
	Family  string `xml:"family,attr"`
	Action  string `xml:"action,attr"`
	Comment string `xml:"comment"`

	Instructions []any `xml:",any"`
}

type Array struct {
	Name string `xml:"name,attr"`
	Type string `xml:"type,attr"`
}

type Break struct{}

type Field struct {
	Name     string `xml:"name,attr"`
	Type     string `xml:"type,attr"`
	Optional bool   `xml:"optional,attr"`
}

which makes using the structs look more straight-forward (for my sensibilities):

var p Protocol
must(xml.Unmarshal([]byte(sXML), &p))
for _, it := range p.Packets[0].Instructions {
	fmt.Printf("%T: %+v\n", it, it)
}

to get:

*main.Array: &{Name:positions Type:NpcUpdatePosition}
*main.Break: &{}
*main.Array: &{Name:attacks Type:NpcUpdateAttack}
*main.Break: &{}
*main.Array: &{Name:chats Type:NpcUpdateChat}
*main.Break: &{}
*main.Field: &{Name:hp Type:short Optional:true}
*main.Field: &{Name:tp Type:short Optional:true}

So, I guess that means that UnmarshalXML must carry the balance of logic and work:

func (p *Packet) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
	for _, attr := range start.Attr {
		switch attr.Name.Local {
		case "family":
			p.Family = attr.Value
		case "action":
			p.Action = attr.Value
		}
	}

	for {
		t, err := d.Token()
		if atEOF(err) {
			break
		}

		if ee, ok := t.(xml.EndElement); ok {
			if ee.Name.Local == start.Name.Local {
				break
			}
		}

		se, ok := t.(xml.StartElement)
		if !ok {
			continue
		}

		if se.Name.Local == "comment" {
			var s string
			must(d.DecodeElement(&s, &se))
			p.Comment = s
			continue
		}

		var dst any
		switch se.Name.Local {
		default:
			continue
		case "array":
			dst = new(Array)
		case "break":
			dst = new(Break)
		case "field":
			dst = new(Field)
		}
		must(d.DecodeElement(dst, &se))

		p.Instructions = append(p.Instructions, dst)
	}

	return nil
}

I still don't understand the implementation notes in the documentation for the xml.Unmarshaler type:

> UnmarshalXML decodes a single XML element beginning with the given start element. If it returns an error, the outer call to Unmarshal stops and returns that error. UnmarshalXML must consume exactly one XML element. One common implementation strategy is to unmarshal into a separate value with a layout matching the expected XML using d.DecodeElement, and then to copy the data from that value into the receiver. Another common strategy is to use d.Token to process the XML object one token at a time. UnmarshalXML may not use d.RawToken.

One thing I learned through trial-and-error was the meaning of 'UnmarshalXML must consume exactly one XML element.'. To satisfy that constraint I added the check to see if the decoder encountered an end element with a name that matches the starting element:

if ee, ok := t.(xml.EndElement); ok {
	if ee.Name.Local == start.Name.Local {
		break
	}
}

though, I now realize this wouldn't work nested elements. A simple depth counter/tracker should clear that up.