The current Apache Pulsar Helmchart loads dashboards from https://github.com/lhotari/pulsar-grafana-dashboards. That's a project that I have created by pulling in dashboards from various sources and cleaned them up to make them usable. These dashboards should be converted to be used together with Pulsar OpenTelemetry metrics. The revisited dashboards could be stored directly in the Apache Pulsar Helmchart project for further maintenance.
Apache Pulsar Helmchart currently uses VictoriaMetrics which natively supports OTLP ingestion. Pulsar and BookKeeper should be configured to push OpenTelemetry metrics directly to the OTLP metrics endpoint.
Besides making the changes, OpenTelemetry should be documented in the Pulsar documentation as the preferred option for metrics. Obviously many customers will continue to use Prometheus.
It would be useful if we could get rid of the duplicate code in Pulsar to expose both Prometheus metrics and OpenTelemetry metrics. However, exposing Prometheus metrics in a different way could be a performance challenge since in large deployments, the uncompressed size of metrics could exceed 500MB. The current /metrics endpoint in Pulsar is heavily optimized for high volumes of metrics.
The metric names would break also when using OpenTelemetry Prometheus exporter, so it would be a major breaking change to remove the existing native Prometheus metrics support in Pulsar.
The current Apache Pulsar Helmchart loads dashboards from https://github.com/lhotari/pulsar-grafana-dashboards. That's a project that I have created by pulling in dashboards from various sources and cleaned them up to make them usable. These dashboards should be converted to be used together with Pulsar OpenTelemetry metrics. The revisited dashboards could be stored directly in the Apache Pulsar Helmchart project for further maintenance.
Apache Pulsar Helmchart currently uses VictoriaMetrics which natively supports OTLP ingestion. Pulsar and BookKeeper should be configured to push OpenTelemetry metrics directly to the OTLP metrics endpoint.
Besides making the changes, OpenTelemetry should be documented in the Pulsar documentation as the preferred option for metrics. Obviously many customers will continue to use Prometheus.
It would be useful if we could get rid of the duplicate code in Pulsar to expose both Prometheus metrics and OpenTelemetry metrics. However, exposing Prometheus metrics in a different way could be a performance challenge since in large deployments, the uncompressed size of metrics could exceed 500MB. The current /metrics endpoint in Pulsar is heavily optimized for high volumes of metrics.
The metric names would break also when using OpenTelemetry Prometheus exporter, so it would be a major breaking change to remove the existing native Prometheus metrics support in Pulsar.