Jenkins + Apache with mod_proxy

I have an AWS EC2 instance running, with only port 22 and 443 as inbound security rules.

Jenkins Configuration

in file -->> /etc/default/jenkins
HTTP_PORT=8080
NAME=jenkins
JENKINS_ARGS="--webroot=/var/cache/$NAME/war --httpPort=$HTTP_PORT --httpListenAddress=127.0.0.1 --prefix=/jenkins"

Apache Configuration

<IfModule mod_ssl.c>
    <VirtualHost *:443>

            ServerAdmin email@domain
            ServerName subdomain.domain.dev
            ServerAlias www.subdomain.domain.dev

            DocumentRoot /var/www/subdomain.domain.dev

            ErrorLog ${APACHE_LOG_DIR}/error.log
            CustomLog ${APACHE_LOG_DIR}/access.log combined

            ProxyPass         /jenkins  http://127.0.0.1:8080/jenkins nocanon
            ProxyPassReverse  /jenkins  http://127.0.0.1:8080/jenkins
            ProxyPassReverse /jenkins http://subdomain.domain.dev/jenkins
            ProxyPassReverse /jenkins https://subdomain.domain.dev/jenkins
            ProxyRequests     Off
            ProxyPreserveHost On
            AllowEncodedSlashes NoDecode

            RequestHeader set X-Forwarded-Proto "https"
            RequestHeader set X-Forwarded-Port "443"

            <Proxy http://127.0.0.1:8080/jenkins*>
                    Order deny,allow
                    Allow from all
            </Proxy>


            Include /etc/letsencrypt/options-ssl-apache.conf        
            SSLCertificateFile /etc/letsencrypt/live/subdomain.domain.dev/fullchain.pem
            SSLCertificateKeyFile /etc/letsencrypt/live/subdomain.domain.dev/privkey.pem

    </VirtualHost>
 </IfModule>

ports.conf

Listen 80

<IfModule ssl_module>
    Listen 443
</IfModule>

<IfModule mod_gnutls.c>
    Listen 443
</IfModule>

With the above configurations, when I try to go to https://subdomain.domain.dev/jenkins it gives a 403 and then redirects to https://subdomain.domain.dev/login?from=%2Fjenkins where it gives a 404 error.

Apache Access Logs

162.158.227.119 - - [20/May/2023:17:09:38 +0000] "GET /jenkins HTTP/1.1" 403 5890 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
162.158.227.119 - - [20/May/2023:17:09:39 +0000] "GET /login?from=%2Fjenkins HTTP/1.1" 404 525 "https://subdomain.domain.dev/jenkins" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"

There are no errors being logged in /var/log/apache2/error.log

Running sudo apache2ctl -t -D DUMP_VHOSTS gives the following output

VirtualHost configuration:
*:443                  subdomain.domain.dev (/etc/apache2/sites-enabled/subdomain.domain.dev-le-ssl.conf:2)

Also, I am using Cloudflare with Full Strict SSL/TLS and used letsencrypt to generate the SSL Certificate.

OpenSSL Version OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

Adding screenshot of the redirect from the browser

Resources

1. Jenkins Official Reverse Proxy Configuration Apache
2. How to Setup Jenkins with SSL with Apache Reverse Proxy on Ubuntu 18.04
3. Install and Configure Apache as Reverse Proxy for Jenkins
4. Apache reverse proxy config with SSL for Jenkins and Sonar

So, I was able to get this working, though I am not sure if this was the actual solution or just a coincidence.
I first stopped apache, then did a sudo apt remove --purge jenkins.
After that, reinstalled Jenkins and updated the /etc/default/jenkins file with the correct prefix and the httpListenAddress which was 127.0.0.1.
Then after a reboot of the instance itself, I was able to get Jenkins up and running. Have been running and configuring Jenkins for almost a day now, no issues found thus far.
In any case, I am not satisfied that I have actually solved anything. If anyone knows the root cause of this, please let me know.