feat: prometheus metrics endpoint on SN and DN nodes

[joaopaulosr95]

Motivation

When running HSDS on Kubernetes there is currently no machine-readable health signal beyond the /info JSON endpoint. This PR adds a standard Prometheus /metrics endpoint to every node so operators can alert on service health, saturation, and storage-backend problems using the usual Prometheus/Grafana stack.

What changed

New /metrics endpoint (hsds/metrics.py), built on the official
prometheus_client library:

• An aiohttp middleware records hsds_http_requests_total{method,status} and an hsds_http_request_duration_seconds histogram for every request.
• A custom collector exposes state the app already tracks, read at scrape time: node readiness (hsds_node_ready), known DN count (hsds_active_dn_count), asyncio task saturation (hsds_tasks_active / hsds_tasks_max), WARN/ERROR log counts, LRU cache stats (items, dirty count, memory used/target per cache), and storage counters (errors, bytes read/written, labelled by backend).
• The default registry adds process_* / python_gc_* metrics for free.
• /metrics is served regardless of node state, so unhealthy nodes remain observable.

Storage stats bug fixes (found while validating the metrics live):

• fileClient._file_stats_increment was missing its final increment statement, so every POSIX storage counter stayed at zero permanently (s3/azure clients have the line; the file client validated the input and returned without counting).
• bytes_in was initialized but never incremented in any backend; it is now recorded on get_object in the file, s3, and azure clients.

Kubernetes manifests: prometheus.io/scrape annotations and named container ports (sn, dn) in the three deployment manifests so annotation-based discovery scrapes the SN and a PodMonitor can target both containers.

Docs: docs/prometheus_metrics.md with the metric reference, the rationale for each signal, a PodMonitor example, and suggested alert rules.

Dependency: prometheus-client (pure Python, no transitive deps) added to pyproject.toml and requirements.txt.

Testing

• New unit tests in tests/unit/metrics_test.py (middleware counting, exposition rendering, cache/storage/node metrics, handler response, and a regression test for the fileClient stats bug), registered in testall.py and collected by pytest in CI.
• Full python testall.py (unit + integration against a standalone server with 4 DNs, posix storage) passes.
• Verified live: scraped /metrics on SN and DN nodes during the integration run; request counters, latency buckets, cache gauges, and storage byte counters all advance as expected.
• flake8 clean under both CI configurations.

[joaopaulosr95]

@jreadey @mattjala flagging you both as you're the top contributors of this project. Yet independent of 446, this is also part of my attempt to make HSDS easier to operate.

[jreadey]

Thanks for your contribution! Having a way to plug in tools like Grafana will be helpful for bother operators of the service and developers (e.g. to assist in determining where bottlenecks are).
I chatted a bit with gemini about the best approaches for adding metrics, and it agreed that prometheus is the best choice, but had some specific guidance (e.g. using PROMETHEUS_MULTIPROC_DIR) that you mind find helpful. You can find the conversation here: https://share.gemini.google/oPr74iW6dkkd.

Also, the original intent of the /info endpoint was to be the path for fetching metrics, but you added a /metrics path. Can /info just be re-purposed? Or would it be better to have just /metrics and deprecate /info?

Finally, how is(should) security handled? Some admins would prefer not to expose HSDS metric data to the outside world. Can the access be restricted to (say) just admin users?