SOLR-18165 Use dot separated metric names#4223
Conversation
There was a problem hiding this comment.
Pull request overview
This PR implements SOLR-18165 by aligning Solr’s OTLP-exported metric names with OpenTelemetry’s dot-separated naming conventions, while keeping Prometheus output stable and preserving Prometheus-style name filtering.
Changes:
- Renames OTLP metric instruments from underscore-separated to dot-separated names across core, node, and CrossDC metrics.
- Updates
FilterablePrometheusMetricReaderto filter using Prometheus base names (getMetadata().getPrometheusName()) rather than OTel internal names. - Adds release-facing documentation: Solr 10.1 upgrade note + an unreleased changelog YAML entry describing the breaking change.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc | Documents the OTLP metric-name breaking change and Prometheus impact/workaround. |
| solr/modules/cross-dc/src/java/org/apache/solr/crossdc/update/processor/ProducerMetrics.java | Updates CrossDC producer OTLP metric names to dot-separated format. |
| solr/core/src/java/org/apache/solr/update/UpdateShardHandler.java | Updates executor metric prefix to dot-separated format. |
| solr/core/src/java/org/apache/solr/update/UpdateLog.java | Updates update log OTLP metric names to dot-separated format (keeping selected underscores). |
| solr/core/src/java/org/apache/solr/update/SolrIndexWriter.java | Updates index writer OTLP metric names to dot-separated format. |
| solr/core/src/java/org/apache/solr/update/PeerSyncWithLeader.java | Updates peer-sync-with-leader OTLP metric names to dot-separated format. |
| solr/core/src/java/org/apache/solr/update/PeerSync.java | Updates peer-sync OTLP metric names to dot-separated format. |
| solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java | Updates update handler OTLP metric names to dot-separated format. |
| solr/core/src/java/org/apache/solr/search/stats/StatsCache.java | Updates stats cache OTLP metric name to dot-separated format. |
| solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java | Updates index searcher OTLP metric names/prefixes to dot-separated format. |
| solr/core/src/java/org/apache/solr/search/SolrFieldCacheBean.java | Updates field cache OTLP metric names to dot-separated format. |
| solr/core/src/java/org/apache/solr/metrics/otel/instruments/AttributedInstrumentFactory.java | Updates core→node metric-name prefix substitution to dot-separated form. |
| solr/core/src/java/org/apache/solr/metrics/otel/FilterablePrometheusMetricReader.java | Switches name filtering to Prometheus base names to keep /admin/metrics?wt=prometheus&name=... working post-rename. |
| solr/core/src/java/org/apache/solr/handler/component/SuggestComponent.java | Updates suggester OTLP metric name to dot-separated format. |
| solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java | Updates HTTP shard executor metric prefix to dot-separated format. |
| solr/core/src/java/org/apache/solr/handler/admin/CoreAdminHandler.java | Updates node executor metric prefix to dot-separated format. |
| solr/core/src/java/org/apache/solr/handler/RequestHandlerBase.java | Updates request handler OTLP metric names to dot-separated format. |
| solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java | Updates replication OTLP metric names to dot-separated format (keeping selected underscores). |
| solr/core/src/java/org/apache/solr/core/SolrCore.java | Updates core/searcher OTLP metric names to dot-separated format. |
| solr/core/src/java/org/apache/solr/core/CoreContainer.java | Updates node cache/executor metric prefixes to dot-separated format. |
| changelog/unreleased/SOLR-18165-dot-separated-metric-names.yml | Adds changelog entry describing the change and its back-compat impact. |
Comments suppressed due to low confidence (1)
solr/core/src/java/org/apache/solr/metrics/otel/FilterablePrometheusMetricReader.java:80
- The suffix-stripping logic only handles
_total,_sum,_bucket,_created, and_info, but Prometheus histograms/summaries also expose a*_countseries. If a user filters withname=<metric>_count, it won't be sanitized to the base name and the metric won't match. Consider adding_countto the recognized suffix set (and keep the doc/examples in sync).
// Users may filter by Prometheus-format names (e.g. "solr_core_requests") or with a
// Prometheus type suffix (e.g. "solr_core_requests_total"). Strip any such suffix so we can
// compare against the Prometheus base name returned by getMetadata().getPrometheusName().
Set<String> sanitizedNames =
includedNames.stream()
.map(
(name) -> {
for (String suffix : PROM_SUFFIXES) {
if (name.endsWith(suffix)) {
return name.substring(0, name.lastIndexOf(suffix));
}
}
return name;
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| @@ -84,7 +85,18 @@ public MetricSnapshots collect( | |||
| if (sanitizedNames.isEmpty()) { | |||
| snapshotsToFilter = super.collect(); | |||
| } else { | |||
| snapshotsToFilter = super.collect(sanitizedNames::contains); | |||
| // We collect all metrics and filter by Prometheus name rather than using | |||
| // super.collect(Predicate) which matches on OTel internal names. This avoids a mismatch | |||
| // when OTel names use dot-separators (e.g. "solr.core.requests") but users filter by the | |||
| // Prometheus underscore-format name they see in the output (e.g. "solr_core_requests"). | |||
| MetricSnapshots all = super.collect(); | |||
| MetricSnapshots.Builder nameFiltered = MetricSnapshots.builder(); | |||
| for (MetricSnapshot snapshot : all) { | |||
| if (sanitizedNames.contains(snapshot.getMetadata().getPrometheusName())) { | |||
| nameFiltered.metricSnapshot(snapshot); | |||
| } | |||
| } | |||
| snapshotsToFilter = nameFiltered.build(); | |||
Co-authored-by: Matthew Biscocho <biscocho.matthew@gmail.com>
solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
Outdated
Show resolved
Hide resolved
Co-authored-by: Chan Chan <chan.dx.dev@gmail.com>
Deprecate solr.core.indexsearcher.open.warmup_time
|
|
||
| === OTEL metrics classified as BETA | ||
|
|
||
| The new OTEL metrics system, naming and dashboard should be treated as a BETA. We reserve the right to make breaking changes in this space in minor versions. |
There was a problem hiding this comment.
I added a BETA declaration her as proposed.
|
|
||
| === Metric solr_core_indexsearcher_open_warmup_time is deprecated | ||
|
|
||
| If you use `solr_core_indexsearcher_open_warmup_time` in a dashboard, it duplicates `solr_core_indexsearcher_warmup_time` and is deprecated for removal in a later 10.x release. |
There was a problem hiding this comment.
I did not remove the duplicate but deprecate it. With the BETA regime in place we can kill this in a minor version.
There was a problem hiding this comment.
In this case, we should also then update the # HELP description in the metric to say deprecated.
Implements SOLR-18165: align Solr's OTLP metric names with the OpenTelemetry semantic conventions, which require dot-separated names.
What changed
OTLP metric names now use dots instead of underscores:
solr_core_requestssolr.core.requestssolr_core_update_committed_opssolr.core.update.committed.opssolr_node_executorsolr.node.executorPrometheus metric names are unchanged. The OTel Prometheus exporter automatically converts dots to underscores, so the
/admin/metrics?wt=prometheusoutput still producessolr_core_requests_total,solr_core_update_committed_opsetc.Metric name filtering (
/admin/metrics?wt=prometheus&name=solr_core_requests) continues to work using Prometheus-format names. TheFilterablePrometheusMetricReadernow filters bygetMetadata().getPrometheusName()(underscore format) rather than against the OTel internal name — which also fixes a pre-existing issue where metrics with unit suffixes likejvm_system_memory_bytescould not be filtered by name.Files changed
"solr_core_*"and"solr_node_*"occurrences)AttributedInstrumentFactory.java—toNodeMetricName()updated to use"solr.core"→"solr.node"substitutionFilterablePrometheusMetricReader.java— name-filter logic updated as described abovechangelog/unreleased/SOLR-18165-dot-separated-metric-names.ymlBreaking change
This is a back-compat break for any OTLP consumer relying on
solr_core_*/solr_node_*metric names from v10.0. Affected users can add ametricstransformprocessor in their OpenTelemetry Collector to rename metrics during transition.Review guidance
All feedback welcome, but one area especially worth a second look:
Metric names that retain underscores. The convention used here is:
solr→core→update→committed.ops)The following metric names intentionally retain underscores:
solr.core.is_leaderis_leaderis a single predicate)solr.core.ref_countsolr.core.disk_spacesolr.core.update.auto_commitssolr.core.update.docs.pending_commitsolr.core.update_log.*(7 metrics)update_logis the tlog component namesolr.core.replication.is_leader,is_follower,is_enabled,is_replicating,is_polling_disabledsolr.core.crossdc.producer.submitted.delete_by_idsolr.core.crossdc.producer.submitted.delete_by_querysolr.core.crossdc.producer.document_sizesolr.core.crossdc.producer.doc_too_large_errorssolr.core.indexsearcher.index.num_docssolr.core.indexsearcher.index.commit_sizesolr.core.indexsearcher.open.warmup_timesolr.core.replication.download_speedsolr.core.replication.downloaded_sizesolr.core.replication.time_elapsedsolr.core.searcher.warming_maxChanging any of these now is cheap; once released each rename becomes another breaking change.
https://issues.apache.org/jira/browse/SOLR-18165