Unmarshal JSON into slice of maps with unique elements

I'm unmarshaling some json files into []map[string]string{}
but very often the source is dirty with many repeated equal objects.

The input looks like:

[{"sa1":"8172"},{"sa3":"8175"},{"sa1":"8172"},{"sa3":"8175"},{"sa3":"8175"},{"sa3":"8175"},{"sa1":"8172"},{"sa3":"8175"},{"sa3":"8175"}]

Resulting in:

map[sa1:8172]
([]map[string]string) (len=9 cap=9) {
 (map[string]string) (len=1) {
  (string) (len=3) "sa1": (string) (len=4) "8172"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa1": (string) (len=4) "8172"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa1": (string) (len=4) "8172"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 },
 (map[string]string) (len=1) {
  (string) (len=3) "sa3": (string) (len=4) "8175"
 }
}

How could I clean the slice of maps to contain only unique elements?

One option is to unmarshal the key value pairs directly into a comparable type, like a struct:

type Elem struct {
	k string
	v string
}

func (e *Elem) UnmarshalJSON(d []byte) error {
	m := map[string]string{}
	if err := json.Unmarshal(d, &m); err != nil {
		return err
	}

	for k, v := range m {
		e.k = k
		e.v = v
		return nil
	}
	return nil
}

Once you can compare the elements individually, you could also wrap that in a collection which filters the elements while unmarshaling. Whether to do this implicitly here, or after the fact is a matter of opinion. It may be a better separation of concerns to make filtering its own method, but I included it in UnmarshalJSON for brevity.

type Elems []Elem

func (e *Elems) UnmarshalJSON(d []byte) error {
	tmp := []Elem{}
	err := json.Unmarshal(d, &tmp)
	if err != nil {
		return err
	}

	seen := map[Elem]bool{}
	for _, elem := range tmp {
		if seen[elem] {
			continue
		}
		seen[elem] = true
		*e = append(*e, elem)
	}
	return nil
}

Then you can unmarshal into Elems:

	elems := Elems{}

	err := json.Unmarshal(js, &elems)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(elems)

Which will give you the two unique pairs: [{sa1 8172} {sa3 8175}]

https://go.dev/play/p/U0iqBAjvz-1