IP Access restriction using Istio not working

I am trying to implement IP-based access restriction using Istio where I would like Istio to block all requests apart from Certain IPs. I got to a point where It's blocking the requests but it's not allowing the IPs I add in the YAML Manifest. I suspect the IP that it is seeing is different that the original Client IP as I have a couple of Load Balancers in front of my Istio.

I read from some documents that X-Forwarder-For is what we should be using to get the actual IP and I have used this in some other implementations but unclear how to get Istio to pick the actual X-Forwarder-Host. Can we create some custom headers and make Istio read from there? Please help

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ingress-policy
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: ALLOW
  rules:
  - from:
    - source:
        remoteIpBlocks: ["98.XXX.XXX.66"]

I enabled Debug Logs in my Envoy and below is what is getting recorded.

[2023-05-13T03:22:05.387Z] "GET /cluster/XXXX/XXXX/hari5 HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "98.XXX.XXX.66:52063,10.240.208.62" "Mozilla/5.0
(Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" "37ccfaa6-7deb-414f-9f5c-72926c997994" "XXX-XX-XXX.XXX.com" "-"
outbound|80||XXX.XXX-XXX.svc.cluster.local - 10.240.208.23:8080 10.240.208.62:40755 - -

> I suspect the IP that it is seeing is different that the original
> Client IP as I have a couple of Load Balancers in front of my Istio.

Assuming you don't use externalTrafficPolicy: Local because as described here you have used remoteIpBlocks. So if you comply with that the only issue I am seeing is the following.

The load balancer is the one component that sits between the client and Istio. Therefore it should be the load balancer that reads the client's IP and adds this value to the header X-Forwarded-For. Then Istio would be able to understand and filter the request based on the Client's Ip address.

This is what we had to use for restricting GET-access based on IP for one of our apps.

The x-forwarded-for header is just a comma-delimited string where first entry is the client IP-address, the remaining IP-addresses are from gateway, proxy etc.

• For match with single IP-address: E.g. "123.234.345.567,*" (note the comma before the wild-card character is important when last segment is less than 3 digits)
• For match with all addresses in last segment: E.g. "123.234.345.*"
• For match with all addresses in 2 last segments: E.g. "123.234.*"

You may also add multiple IP-patterns under values.

I hope this will bring you further.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: our-authorization-policy
  namespace: our-namespace
spec:
  selector:
    matchLabels:
      app: our-app-name
  action: ALLOW
  rules:
    - to:
        - operation:
            methods: ["GET"]
      when:
        - key: request.headers[x-forwarded-for]
          values:
            - "xxx.xxx.xxx.xxx,*" # note the trailing '*'