Computed error bars do not plot for factor data in ggplot2 R

I am trying to plot error bars for factor data as shown in this reproducible example here. This example just runs fine. However, I tried the same thing with my original dataset, but errorbars refuse to plot. I am a little lost. The following are the relevant pieces of code and as reproducible as possible. Any idea, why I a cannot get the errorr bars to work?

Dataframe df has two columns: yaxis.og and species (4 species) like this:

 yaxis.og species
      <dbl> <fct>  
 1   2.56   A      
 2   3.81   B      
 3   1.48   C      
 4   1.65   D      
 5  -0.177  A  
      .....

I computed a simple linear model and stored the results as such:

summary_df <- summary(mod)$coefficients %>% 
  as_tibble() %>% 
  dplyr::mutate(species = c("A","B","C","D")) %>%
  dplyr::rename(yaxis = Estimate) %>%
  mutate(ymin = yaxis - 1.96 * `Std. Error`,
         ymax = yaxis + 1.96 * `Std. Error`)


yaxis `Std. Error`    df `t value` `Pr(>|t|)` species  ymin  ymax
  <dbl>        <dbl> <dbl>     <dbl>      <dbl> <chr>   <dbl> <dbl>
1 0.611        0.145  61.2      4.22   8.21e- 5 A       0.327 0.895
2 2.16         0.160  78.0     13.5    3.93e-22 B       1.85  2.47 
3 1.21         0.153  68.7      7.92   2.84e-11 C       0.912 1.51 
4 2.21         0.223 162.       9.89   2.43e-18 D       1.77  2.64 

When I plot it without erorrbars, it plots fine:

p <- ggplot(aes(x = species, y = yaxis.og, color = species, group = species), data = df) + 
geom_jitter(width = 0.1)  +
geom_point(aes(x = species, y = yaxis), color = 'black', size = 2, data = summary_df) 

When I add errorbars in, it throws a huge error. The error doesn't make to much sense either because the last layer uses a different dataset, so why would it need yaxis.og?

p + geom_errorbar(aes(x = species, ymin = ymin, ymax = ymax), data = summary_df)

Error in `geom_errorbar()`:
! Problem while computing aesthetics.
ℹ Error occurred in the 3rd layer.
Caused by error in `FUN()`:
! object 'yaxis.og' not found
Backtrace:
  1. base (local) `<fn>`(x)
  2. ggplot2:::print.ggplot(x)
  4. ggplot2:::ggplot_build.ggplot(x)
  5. ggplot2:::by_layer(...)
 12. ggplot2 (local) f(l = layers[[i]], d = data[[i]])
 13. l$compute_aesthetics(d, plot)
 14. ggplot2 (local) compute_aesthetics(..., self = self)
 15. base::lapply(aesthetics, eval_tidy, data = data, env = env)
 16. rlang (local) FUN(X[[i]], ...)

The issue is that all aes defined in ggplot() are global aesthetics and will inherited by all geom layers even if not used. As a result ggplot2 expects a column with name yaxis.og to be present in the data used by each geom layer, except in case you override the global aes as you do e.g. in geom_point.

One option to fix that would be add inherit.aes=FALSE to geom_errorbar which prevents the global aes to be inherited or to override the global aes by using aes(..., y = NULL).

Using some fake example data based on iris:

library(ggplot2)

p +
  geom_errorbar(
    aes(x = species, ymin = ymin, ymax = ymax, y = NULL),
    data = summary_df
  )

<!-- -->

p +
  geom_errorbar(
    aes(x = species, ymin = ymin, ymax = ymax),
    data = summary_df, inherit.aes = FALSE
  )

<!-- -->

DATA

set.seed(123)

df <- data.frame(
  yaxis.og = c(iris$Sepal.Length, sample(iris$Sepal.Length, 50)),
  species = rep(c("A", "B", "C", "D"), each = 50)
)

mod <- lm(yaxis.og ~ species - 1, data = df)