feat(prometheus): add Clouseau, DB count, IOQ and Smoosh metrics

[smithsz]

Overview

Expose new Prometheus metrics for monitoring:

• Clouseau search connectivity status
• Total database count in cluster
• IOQ active channels and request counts
• Smoosh compaction job statistics (active/starting/waiting)

Improves observability of core CouchDB subsystems.

Testing recommendations

./dev/run

curl http://localhost:15984/_node/_local/_prometheus | grep -E '^couchdb_(clouseau_connected|database_count|ioq_|smoosh_)'

Example

# HELP couchdb_clouseau_connected clouseau connectivity status
# TYPE couchdb_clouseau_connected gauge
couchdb_clouseau_connected 0

# HELP couchdb_database_count total database count
# TYPE couchdb_database_count gauge
couchdb_database_count 2

# HELP couchdb_ioq_interactive_requests IOQ interactive queue requests
# TYPE couchdb_ioq_interactive_requests gauge
couchdb_ioq_interactive_requests 0

# HELP couchdb_ioq_background_requests IOQ background queue requests
# TYPE couchdb_ioq_background_requests gauge
couchdb_ioq_background_requests 0

# HELP couchdb_ioq_total_requests IOQ total active requests
# TYPE couchdb_ioq_total_requests gauge
couchdb_ioq_total_requests 0

# HELP couchdb_smoosh_channel_count total active smoosh channels
# TYPE couchdb_smoosh_channel_count gauge
couchdb_smoosh_channel_count 7

# HELP couchdb_smoosh_active_jobs global total active compaction jobs
# TYPE couchdb_smoosh_active_jobs gauge
couchdb_smoosh_active_jobs 0

# HELP couchdb_smoosh_starting_jobs global total starting compaction jobs
# TYPE couchdb_smoosh_starting_jobs gauge
couchdb_smoosh_starting_jobs 0

# HELP couchdb_smoosh_waiting_jobs global total waiting compaction jobs
# TYPE couchdb_smoosh_waiting_jobs gauge
couchdb_smoosh_waiting_jobs 0

Checklist

• This is my own work, I did not use AI, LLM's or similar technology
• Code is written and works correctly
• Changes are covered by tests
• Any new configurable parameters are documented in rel/overlay/etc/default.ini
• Documentation changes were made in the src/docs folder
• Documentation changes were backported (separated PR) to affected branches

[rnewson]

function does not exist in couchdb's ioq implementation.

Oddly, https://github.com/apache/couchdb-ioq is where the "real" IOQ is (the one Cloudant uses and maintains) but it is not in use by couchdb.

Perhaps now is the time to fix that.

[smithsz]

Since switching over is a bit more effort, I went ahead and updated the PR to support both IOQ implementations for now. Hope that's alright.

[nickva]

One issue with this ioq is it's almost too wide spreading N queues over N schedulers that it effectively becomes a bypass. Ideally we'd have a little bit of a queue to do the prioritization against. Maybe we could some kind of a configurable limit or middle ground have 8 or 16 IO queues even if we have 80 schedulers...

[nickva]

Thanks for the contribution, Sam!

Looks good overall the main thing we'd have to fix is the lack of ioq settings and the missing call in Apache CouchDB. You're using the alternative IOQ implementation https://github.com/apache/couchdb-ioq with multiple channels and the ioq:get_disk_queues() call. That function is not present in the built-in Apache CouchDB ioq.

The lack of ioq:get_disk_queues() is why the CI fails right off the bat. The xref CI step detects the unknown function call.

So, we'd have to make your PR work with both the built-in IOQ and the fancier one https://github.com/apache/couchdb-ioq. That could be done by adding a proxy ioq:get_disk_queues() function to ioq.erl and make it call get_queue_lengths(). Then we'd have to add the channels from get_queue_lengths() in the built-in ioq to the metrics. CouchDB users will want to see those most of all. That's just background vs interactive so add those to the prometheus metrics and in the tests for it.

If you add the proxy ioq:get_disk_queues() we can also clean up the xref exceptions in https://github.com/apache/couchdb/blob/main/test/fixtures/allowed-xref.txt

1:src/ioq.erl: Warning: ioq:get_disk_queues/0 is undefined function (Xref)
2:src/weatherreport_check_ioq.erl:{95,1}: Warning: weatherreport_check_ioq:check_legacy_int/1 calls undefined function ioq:get_disk_queues/0 (Xref)

[nickva]

Oops, I hadn't realized they actually return different data types maps vs kv lists. Instead of having to add handlers for both on every call let's instead make get_disk_queues() return a kv list. It would be as simple as passing it through maps:to_list(Map). For example:

> maps:to_list(#{a=>1}).
[{a,1}]

[nickva]

+1