A setting to subjectively adjust random effect variance in R package glmmTMB?

From my layman understanding of frequentist hierarchical models, there is some penalty mechanism built into the likelihood function, that prevents the random effects from overfitting to the data and instead ‘shrinks’ them towards a group mean.

Apparently, this penalty mechanism can sometimes be adjusted by the modeller to reflect their belief that there should be more/less variance between groups’ random effects than is achieved with the likelihood function’s default setting.

Is this an option with glmmTMB in R? Is there a setting/parameter than can be adjusted to regulate the variance attributed to random effects?

Yes, you can set the random effects variance to whatever you want by using the map argument to specify which model parameters should be fixed rather than estimated (and start to specify their values).

To illustrate, start by fitting the model (a model with only a single random effects variance, for simplicity) to see what the data (+ assumptions) say the variance "should" be:

library(glmmTMB)
data("sleepstudy", package = "lme4")
m1 <- glmmTMB(Reaction ~ Days + (1|Subject), sleepstudy)
VarCorr(m1)

Conditional model:
 Groups   Name        Std.Dev.
 Subject  (Intercept) 36.012  
 Residual             30.895  

Now suppose we think the model has (for some reason) overestimated the among-subject variance, i.e. the subject intercepts "should" be more tightly grouped.

We need to know that the random effects are parameterized on a log-SD scale, and that the relevant parameter is called theta (there is a little bit of this information in the "Mapping" section of the covstruct vignette).

m2 <- update(m1, 
             map = list(theta = factor(NA)), 
             start = list(theta = log(10)))

(You don't have to use update() - you could have fitted the model from scratch with these arguments.) Check that this worked:

Conditional model:
 Groups   Name        Std.Dev.
 Subject  (Intercept) 10.000  
 Residual             37.248  

You could look at the fixed effects and their SEs etc. to see how this choice affects the rest of the model.