15692 metrics port hung

[howardlee2010]

Describe the bug

I have a three-node RabbitMQ cluster deployed via an operator, with RabbitMQ version 3.12.2. Right after the cluster is deployed, running the command curl http://:15692/metrics returns data successfully. However, after some time (ranging from tens of minutes to several hours), executing the same command will hang. After about one minute, it throws the error curl: (52) Empty reply from server. The detailed output is as follows:

curl -vk http://10.239.108.77:15692/metrics

• Trying 10.239.108.77:15692...
• Connected to 10.239.108.77 (10.239.108.77) port 15692 (#0)

GET /metrics HTTP/1.1
Host: 10.239.108.77:15692
User-Agent: curl/7.71.1
Accept: /

• Empty reply from server
• Connection #0 to host 10.239.108.77 left intact
  curl: (52) Empty reply from server

At this point, logging into the corresponding pod and running the command rabbitmq-diagnostics runtime_thread_stats also hangs with no output whatsoever. Does anyone know the root cause of this issue? It has been troubling me for days. Thanks to anyone who can offer help.

Reproduction steps

1.curl http://:15692/metrics
2.
3.
...

Expected behavior

Does anyone know the root cause of this issue? It has been troubling me for days. Thanks to anyone who can offer help.

Additional context

No response

[lukebakken]

with RabbitMQ version 3.12.2

Please read the community support policy: link

Your very brief description, with no additional artifacts for us to investigate, is basically asking us to guess. So, my guess is that you have something very environment-specific - if what you report were common, we would hear about it many times, every day.

[michaelklishin]

@howardlee2010 RabbitMQ 3.12 is entirely out of support.

While the Prometheus endpoints can produce very large result sets, rabbitmq-diagnostics "hanging" can mean anything from a firewall blocking its connection to the node (or epmd first) to the node experiencing very severe CPU resource starvation, to many things in between.

Two well known scenarios that can exhibit similar behavior:

1. RabbitMQ nodes that are given like 0.5 of one CPU core. That's not a reasonable number of cores for distributed messaging infrastructure nodes
2. Something in your system declares a classic queue (or a set of classic queues) with millions of priorities. These days the maximum number of priorities should be capped at a few hundreds but if you combine it with underprovisioned CPU resources, that can still be enough to make a visible difference

Use OS resource monitoring to narrow down the problem.

Not only we will not guess what may be going on with your node, we won't provide any more guidance on 3.12, which is entirely out of support, including for paying and regularly contributing users.