Trim white space in ggplot modified raincloud plot with discrete variables

I'm having difficulty trimming white space from the discrete axis of a ggplot. See reprex below - how can I remove the white space between the data for "large" and the x-axis? I've tried various versions of scale_x_discrete but it doesn't seem to have any effect.

# Prepare data
sample_data <- iris %>%
    filter(Species == "setosa" | Species == "virginica") %>%
    mutate(category = case_when(Sepal.Length > 5 ~ "large",
                                Sepal.Length <= 5 ~ "small")) %>%
    select(Sepal.Width,
           Species,
           category)
    
# Figure
ggplot(data = sample_data) +
    aes(x = category,
        y = Sepal.Width,
        fill = category) +
    ggdist::stat_halfeye(
        adjust = 0.5,
        justification = -.1,
        .width = 0,
        point_colour = NA
    ) +
    geom_boxplot(width = .05,
                 outlier.colour = NA,
                 alpha = 0.5) +
    ggdist::stat_dots(
        side = "left",
        justification = 1.1,
        binwidth = 0.015) +
    facet_grid(~ Species) +
    labs(title = "Figure 1: Change in sepal width with sepal length",
         x = "Sepal length") +
    coord_flip() +
    tidyquant::scale_fill_tq() +
    tidyquant::theme_tq()

One option to trim the whitespace would be to set the limits for the x scale via the xlim argument of coord_flip. Note however that setting the lower bound for the limits will require some fiddling and depends on the plot size or height.

library(ggplot2)
library(ggdist)
library(tidyquant)

ggplot(data = sample_data) +
  aes(
    x = category,
    y = Sepal.Width,
    fill = category
  ) +
  ggdist::stat_halfeye(
    adjust = 0.5,
    justification = -.1,
    .width = 0,
    point_colour = NA
  ) +
  geom_boxplot(
    width = .05,
    outlier.colour = NA,
    alpha = 0.5
  ) +
  ggdist::stat_dots(
    side = "left",
    justification = 1.1,
    binwidth = 0.015
  ) +
  facet_grid(~Species) +
  labs(
    title = "Figure 1: Change in sepal width with sepal length",
    x = "Sepal length"
  ) +
  tidyquant::scale_fill_tq() +
  tidyquant::theme_tq() +
  coord_flip(xlim = c(1.25, NA))