mapstructure decode a map[string]interface{} into a struct containing custom type gives Error

My app takes a JSON string, unmarshal into a struct BiggerType and then, using mapstructure to decode the struct's field Settings into a type MainType. BiggerType needs to support multiple types of Settings and hence, is and has to be declared as map[string]interface{}. The app used to be working fine until we have a new type of Settings, i.e. MainType containing some custom fields with the type SpecialType.

The structs and the main codes are included below. Running the code gives the following error.

* 'b' expected a map, got 'string'

Some codes were removed for brevity

package main

import (
	...
	"github.com/mitchellh/mapstructure"
)

const myJSON = `{
  "settings": {
    "a": {
      "aa": {
        "aaa": {
          "sta": "special_object_A",
          "stb": "special_object_B"
        },
        "aab": "bab"
      },
      "ab": true
    },
    "b": "special_string"
  },
  "other": "other"
}`

func main() {
	var biggerType BiggerType

	err := json.Unmarshal([]byte(myJSON), &biggerType)
	if err != nil {
		panic(err)
	}

	var decodedMainType MainType

	if err := mapstructure.Decode(biggerType.Settings, &decodedMainType); err != nil {
		panic(err)
	}
}

type BiggerType struct {
	Other    string   `json:"other"`
	// Settings MainType `json:"settings"` can't be used as it needs to support other "Settings"
	Settings map[string]interface{} `json:"settings"`
}

type A struct {
	Aa *AA   `json:"aa"`
	Ab *bool `json:"ab"`
}

type AA struct {
	Aaa SpecialType `json:"aaa"`
	Aab string      `json:"aab"`
}

type MainType struct {
	A A           `json:"a"`
	B SpecialType `json:"b"`
}

type SpecialTypeObject struct {
	Sta string `json:"sta"`
	Stb string `json:"stb"`
}

func (s SpecialTypeObject) InterfaceMethod() (string, error) {
	return s.Sta + "+" + s.Stb, nil
}

type SpecialTypeString string

func (s SpecialTypeString) InterfaceMethod() (string, error) {
	return string(s), nil
}

type SpecialInterface interface {
	InterfaceMethod() (string, error)
}

// SpecialType SpecialTypeString | SpecialTypeObject
type SpecialType struct {
	Value SpecialInterface
}

func (s *SpecialType) UnmarshalJSON(data []byte) error {
	...
}

My goal is to be able to decode biggerType.Settings into decodedMainType with all the values intact. Can anyone please share with me any workaround or/and suggestions?

The playground to replicate the issue: https://go.dev/play/p/G6mdnVoE2vZ

Thank you.

I'm with Daniel Farrell on this, but it's possible to fix the existing solution.

The idea is to make use of the mapstructure's ability to customize its Decoder which can take a "hook" which is called at key points during the decoding process. There are multiple available hooks and they can be combined into a "chain-style" hook.

Since there's a stock hook which is able to check whether the target variable's type implements encoding.TextUnmarshaler, we can tweak your (*SpecialType).UnmarshalJSON to become (*SpecialType).UnmarshalText and then use that:

func (s *SpecialType) UnmarshalText(data []byte) error {
	if len(data) > 0 && data[0] != '{' {
		s.Value = SpecialTypeString(data)
		return nil
	}

	var obj SpecialTypeObject
	if err := json.Unmarshal(data, &obj); err != nil {
		return fmt.Errorf("failed to unmarshal SpecialType: %s", err)
	}
	s.Value = obj
	return nil
}

and

var decodedMainType MainType

msd, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{
	DecodeHook: mapstructure.TextUnmarshallerHookFunc(),
	Result:     &decodedMainType,
})
if err != nil {
	return BiggerType{}, err
}

if err := msd.Decode(biggerType.Settings); err != nil {
	return BiggerType{}, err
}

The complete solution: <https://go.dev/play/p/jaCD9FdSECz>.

But then again — as Daniel hinted at, you appear to have fallen for using mapstructure because it promised to be a no-brainer solution for a problem which appeared to be complicated.
In fact, encoding/json appears to have stock facilities to cover your use case — exactly through making certain types implement json.Unmarshaler, so I'd try to ditch mapstructure for this case: it's a useful package, but now you appear to be working around the rules of the framework it implements.

Why it doesn't work

> * 'b' expected a map, got 'string'

In the JSON, settings.b is a string type. You've decoded it into a BiggerType in which settings is a map[string]interface{}. The standard library provided JSON Unmarshalling process results in a map like:

map[string]interface{} {
   "A": map[string]interface{}{...},
   "B": string("special_value"),
}

JSON decoding is now done. No more Json decoding happens in the program.

You then attempt to use mapstructure to Decode your settings map into a MainType, where B is of type SettingsType. So mapstructure.Decode dutifully iterates over your BiggerType and attempts to turn it into a MainType, but hits a snag. when it gets to B, the input is a string, and the output is a SettingsType.

You wrote an UnmarshalJSON function for settings type, but B's value is not being unmarshalled from JSON - it's already a string. Decode doesn't know how to turn a string into a SpecialType ( it says it expected a map but that's a little misleading as it actually checks for map, struct, slice, and array types)

Rephrasing the problem you're facing

So then let's identify the error a little more specifically: You have. map[string]interface{} with a string value, mapstructure.Decode tries to decode it into a struct field with a Struct value.

Rephrasing the goal

Everything you've shown here is issues you're facing with your solution. What is your underlying problem? You have an interface{} in BiggerType.Settings that might be a string and might be a map[string]interface{}, and you want to turn it into a SpecialType that holds either a SpecialTypeString or a SpecialTypeObj

I can't entirely square that with your question though, where you keep talking about MainType being possibly different types, but all the different types in your program seem to be under SpecialType.

How to Solve

It seems like the obvious solution here is to ditch mapstructure and use the method you already use for SpecialType with a custom unmarshaller. mapstructure is useful for cases when you need to unmarshal JSON to introspect some attributes and then make a decision about how to handle other attributes. You're not doing anything like that with mapstructure though; the closest thing is your custom unmarshaller. If a custom Unmarshaller can do the job, you don't need mapstructure.

SpecialType.UnmarshalJSON in your code is already an example of how you could wrap an unknown value in a known type and use a switch statement to handle unknown value types.

So just ditch mapstructure, redefine BiggerType as:

type BiggerType struct {
	Other    string   `json:"other"`
	Settings MainType `json:"settings"`
}

And let the already-written UnmarshalJSON handle MainType.B.

If it's not SpecialType that needs this logic, but MainType, then do the same thing with SpecialType that you did to MainType.

https://go.dev/play/p/DW8N8t3Lr0l

If you want to see a solution with mapstructure, please provide more than one example of the JSON input so I can see how they differ and why mapstructure might be required.