Enabling Prometheus metrics on WSO2 Micro-Integrator

I have a MI instance 4.1.0 with a port offset of 20 running on RHEL VM (no containerizaion). The MI instance is listening on port 9211

$ grep offset deployment.toml
offset  = 20
$ netstat -tulpn  | grep java
tcp        0      0 0.0.0.0:8300            0.0.0.0:*               LISTEN      18874/java
tcp        0      0 0.0.0.0:9174            0.0.0.0:*               LISTEN      18874/java
tcp        0      0 0.0.0.0:9211            0.0.0.0:*               LISTEN      18874/java
tcp        0      0 0.0.0.0:35932           0.0.0.0:*               LISTEN      18874/java
tcp        0      0 0.0.0.0:8263            0.0.0.0:*               LISTEN      18874/java

When starting MI as follows and checking with curl

$ ./micro-integrator.sh -DenablePrometheusApi
$ curl -v http://XXX:9211/metric-service/metrics
		* About to connect() to XXX port 9211 (#0)
		> GET /metric-service/metrics HTTP/1.1
		> User-Agent: curl/7.29.0
		> Host: XXX:9211
		< HTTP/1.1 200 OK
		< Host: git-chqbiz01:9211
		< Content-Type: text/plain; charset=UTF-8
		< Date: Mon, 22 May 2023 07:45:48 GMT
		< Transfer-Encoding: chunked
		<

There is not a single metrics being reported (reply is empty).
No error messages appear in the logfile

Shouldn't I see the list of default metrics? Is there an additional configuration required? I simply followed the instructions of following link : https://docs.wso2.com/display/EI650/Monitoring+WSO2+Micro+Integrator

If I alter the micro-integrator.sh to add a -DenablePrometheusApi and launch micro-integrator.sh, then I get a HTTP 404 with following log lines.

2023-05-22 15:58:15,722]  WARN {InternalAPIDispatcher} - No Internal API found to dispatch the message to : /metric-service/metrics
[2023-05-22 15:58:21,375]  WARN {InternalAPIDispatcher} - No Internal API found to dispatch the message to : /metric-service/metrics
[2023-05-22 15:58:23,281]  WARN {InternalAPIDispatcher} - No Internal API found to dispatch the message to : /metric-service/metrics
[2023-05-22 15:58:39,322]  WARN {InternalAPIDispatcher} - No Internal API found to dispatch the message to : /metric-service/metrics

Have you added the custom handlers to deployment.toml? And then started the server with -DenablePrometheusApi=true flag?

[[synapse_handlers]]
name="CustomObservabilityHandler"
class="org.wso2.micro.integrator.observability.metric.handler.MetricHandler"

This is the latest document.

If you want to add this flag permanently you can add this to integrator.sh as shown below.

while [ "$status" = "$START_EXIT_STATUS" ]
do
    $JAVACMD \
    -Xbootclasspath/a:"$CARBON_XBOOTCLASSPATH" \
    $JVM_MEM_OPTS \
    -XX:+HeapDumpOnOutOfMemoryError \
    -XX:HeapDumpPath="$CARBON_HOME/repository/logs/heap-dump.hprof" \
    $JAVA_OPTS \
    -Dcom.sun.management.jmxremote \
    -classpath "$CARBON_CLASSPATH" \
    -Djava.io.tmpdir="$CARBON_HOME/tmp" \
    -Dcatalina.base="$CARBON_HOME/wso2/lib/tomcat" \
    -Dwso2.server.standalone=true \
    -Dcarbon.registry.root=/ \
    -Djava.command="$JAVACMD" \
    -Dqpid.conf="/conf/advanced/" \
    $JAVA_VER_BASED_OPTS \
    -Dcarbon.home="$CARBON_HOME" \
    -Dlogger.server.name="micro-integrator" \
    -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
    -Dcarbon.config.dir.path="$CARBON_HOME/conf" \
    -Dcarbon.repository.dir.path="$CARBON_HOME/repository" \
    -Dcarbon.components.dir.path="$CARBON_HOME/wso2/components" \
    -Dcarbon.dropins.dir.path="$CARBON_HOME/dropins" \
    -Dcarbon.external.lib.dir.path="$CARBON_HOME/lib" \
    -Dcarbon.patches.dir.path="$CARBON_HOME/patches" \
    -Dcarbon.internal.lib.dir.path="$CARBON_HOME/wso2/lib" \
    -Dei.extendedURIBasedDispatcher=org.wso2.micro.integrator.core.handlers.IntegratorStatefulHandler \
    -Djava.util.logging.config.file="$CARBON_HOME/conf/etc/logging-bridge.properties" \
    -Dcomponents.repo="$CARBON_HOME/wso2/components/plugins" \
    -Dconf.location="$CARBON_HOME/conf" \
    -Dcom.atomikos.icatch.file="$CARBON_HOME/wso2/lib/transactions.properties" \
    -Dcom.atomikos.icatch.hide_init_file_path=true \
    -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false \
    -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true \
    -Dcom.sun.jndi.ldap.connect.pool.authentication=simple  \
    -Dcom.sun.jndi.ldap.connect.pool.timeout=3000  \
    -Dorg.terracotta.quartz.skipUpdateCheck=true \
    -Djava.security.egd=file:/dev/./urandom \
    -Dfile.encoding=UTF8 \
    -Djava.net.preferIPv4Stack=true \
    -DNonRegistryMode=true \
    -DNonUserCoreMode=true \
    -Dcom.ibm.cacheLocalHost=true \
    -Dcarbon.use.registry.repo=false \
    -DworkerNode=false \
    -Dorg.apache.cxf.io.CachedOutputStream.Threshold=104857600 \
    -Djavax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom=net.sf.saxon.xpath.XPathFactoryImpl \
    -DavoidConfigHashRead=true \
    -Dproperties.file.path=default \
    -DenableReadinessProbe=true \
    -DenableManagementApi=true \
    -Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector \
    $NODE_PARAMS \
    -Dorg.apache.activemq.SERIALIZABLE_PACKAGES="*" \
    -Dnashorn.args=--language=es6 \
    -DenablePrometheusApi=true \
    org.wso2.micro.integrator.bootstrap.Bootstrap $*
    status=$?
done