Jackson: How to use a custom deserializer with @JsonAnySetter annotation?

I have several YAML config files I want to deserialize into a class. The YAML in the files consists of simple name value pairs with no nesting. There's a handful of properties that will have explicit fields, but the rest I just want dumped into a Map.

This all works fine, but I also want all the values of the properties that get deserialized into the Map through .add() to be run through a custom deserializer. I've tried using @JsonDeserialize on the setter value parameter and the setter method itself but Jackson seems to ignore it altogether.

Here's how it's set up:

public class ConfigData {
    private Map<String, Object> dynamicConfig = new LinkedHashMap<>();
    @JsonAnyGetter
    public Map<String, Object> getConfig() {
        return dynamicConfig;
    }
    @JsonAnySetter
    public void add(String name, @JsonDeserialize(using = FooDeserializer.class) Object value) {
        dynamicConfig.put(name, value);
    }
    @JsonProperty("some_special_property")
    public String setSomeSpecialProperty(String value) {
        add("some_special_property", value);
    }
}

And to deserialize:

public static ConfigData getConfig(URL configResource) throws IOException {
    try (InputStream stream = configResource.openStream()) {
        ObjectMapper mapper = new YAMLMapper();
        return mapper.readValue(new InputStreamReader(stream, StandardCharsets.UTF_8), ConfigData.class);
    }
}

I discovered the problem was that I was specifying the deserializer class with the using property of the @JsonDeserialize annotation. For this specific use case I needed to use the contentUsing property instead, which is used for things like the value field of a Map entry.

This is what my setter looks like now:

@JsonAnySetter
@JsonDeserialize(contentUsing = FooDeserializer.class)
public void add(String name, Object value) {
    dynamicConfig.put(name, value);
}

Now all the values will be serialized using FooDeserializer, except for "some_special_property" which has its own setter.