Prometheus Alert rule not showing up

I created alert rule file.yml and copy it under "/" of prometheus contianer and I added it to the values.yml file but I can't see the rule in the UI

    prometheus.yml:
    rule_files:
      - /etc/config/recording_rules.yml
      - /etc/config/alerting_rules.yml
      - /custom_alerting_rules.yml
    ## Below two files are DEPRECATED will be removed from this default values file
      - /etc/config/rules
      - /etc/config/alerts

    alerting:
      alertmanagers:
        - static_configs:
            - targets: ['alertmanager:9093'] here i tried the @IP of alert manager service 

here is the alert file

groups:
  - name: my-custom-alerts
    rules:
      - alert: HighPodCount
        expr: count(kube_pod_info{pod=~"consumer.*"}) > 2
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: High pod count
          description: The number of pods is above the threshold.

k get svc shows

prometheus-alertmanager               ClusterIP      10.10x.21x.x8    <none>        9093/TCP                       68m

what am doing wrong ?

From the perspective of Prometheus server, its configuration and rule files are not auto-reloaded. You need to call the Reload management api manually.

There are 3 ways for this purpose:

1. If you're in the prometheus contianer:

   curl -XPOST http://:::9090/-/reload
   

2. If you're in another pod container of the same namespace:

   # prometheus pod name: prometheus-0
   curl -XPOST http://prometheus-0:9090/-/reload
   

3. If you're out of K8s cluster(i.e. where you run kubectl command):

   # prometheus pod name: prometheus-0, namespace: monitoring
   kubectl port-forward prometheus-0 -nmonitoring 9090:9090
   # in another shell
   curl -XPOST http://127.0.0.1:9090/-/reload
   

If you used prometheus operator you could alternatively use PrometheusRule crd rather than a file :

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-custom-alert
  namespace: my-namespace
spec:
  groups:
  - name: my-custom-alerts
    rules:
      - alert: HighPodCount
        expr: count(kube_pod_info{pod=~"consumer.*"}) > 2
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: High pod count
          description: The number of pods is above the threshold.

Then :

#> kubectl get PrometheusRule -n my-namespace
my-custom-alert

This would have the operator mount the alert within prometheus' config without you having to do anything else.