Traefik is forwarding request to wrong host for a service

I've come across a snag and hope some experts from here can help out.

We are trying to deploy a new service but Traefik is returning 404. And the root cause seems to be it is holding onto a staled host IP for the service. Might be something silly got missed out...

The setup is we have a ECS cluster fronted by Traefik as the ingresser/reverse proxy. We have a new service deployed with following docker labels:

"dockerLabels": {
                "traefik.enable": "true",
                "traefik.frontend.entryPoints": "http",
                "traefik.frontend.passHostHeader": "true",
                "traefik.frontend.priority": "6",
                "traefik.frontend.rule": "Headers: CS-Forwarded-Host,report-api.dev.some-domain.com"

The ECS cluster has got 2 nodes with services deployed across them, traefik is running on both of them. When the service is updated, the traefik logs reveals that it's getting the configuration updated accordingly:

Configuration received from provider ecs:

"backend-service-reporting-public-api-development-python": {
            "servers": {
                "server-service-reporting-public-api-development-python-4405ef23c5c6": {
                    "url": "http://172.30.3.147:57138",
                    "weight": 1
                }
            },
            "loadBalancer": {
                "method": "wrr"
            }
        },

...

"frontend-service-reporting-public-api-development-python": {
            "entryPoints": [
                "http"
            ],
            "backend": "backend-service-reporting-public-api-development-python",
            "routes": {
                "route-frontend-service-reporting-public-api-development-python": {
                    "rule": "Headers: CS-Forwarded-Host,report-api.dev.some-domain.com"
                }
            },
            "passHostHeader": true,
            "priority": 6,
            "basicAuth": []
        },

We have verified it by jumping on the ECS instance and do a curl http://172.30.3.147:57138/swagger/doc which works ok.

However, when hitting the endpoint from outside we always get 404. The following traefik logs reveals it's forwarding requests to the wrong host IP/port, possibly a staled one:

vulcand/oxy/forward: completed ServeHttp on request" Request="
{
    "Method": "GET",
    "URL": {
        "Scheme": "http",
        "Opaque": "",
        "User": null,
        "Host": "172.30.3.147:49181",
        "Path": "",
        "RawPath": "",
        "ForceQuery": false,
        "RawQuery": "",
        "Fragment": ""
    },
    "Proto": "HTTP/1.0",
    "ProtoMajor": 1,
    "ProtoMinor": 0,
    "Header": {
        ...
        "Cs-Forwarded-Host": [
            "report-api.dev.some-domain.com"
        ],
        ...
    }
}"

So instead of forwarding to the latest backend-service-reporting-public-api-development-python url 172.30.3.147:57138, it forwards traffic to 172.30.3.147:49181. Notice the IP is the same but different port number. This obviously will result in a 404. But why traefik is NOT forwarding the requests to the correct URL? I didn't find a way to force traefik to update it's config as this should all be dynamic if my understanding is correct. What am I missing here?

The traefik version is v1.6.6. I know it's ancient. This is on our radar, will hopefully upgrade soon.

Please help out. Thanks in advance.

I solved this mystery finally, thought would share my findings here in case it'd help anyone.

There is a traefik rule set on another service in the ECS cluster (say service x):

"traefik.frontend.rule": "HeadersRegexp: CS-Forwarded-Host,api(-development)?.dev.some-domain.com;PathPrefix:/v1,/v2"

The HeadersRegexp is the culprit. It's wide open matching criteria affected reporting public api service with the custom domain set to report-api.dev.some-domain.com. Since /api(-development)?. dev.some-domain.com/g will match report-api.dev.some-domain.com so the handling of reporting public api service request is hijacked by service x. Service x certainly doesn't have a handler for the request hence 404 returned. The remedy to this is to restrict the regex to ^api(-development)?.dev.some-domain.com.

Quite unfortunate really. Hope this will help someone at some point.