[v8r0] fix (WMS): do not report non-finite job parameters - #8734
Open
aldbr wants to merge 1 commit into
Open
Conversation
Since DIRACGrid#6938 the Watchdog reports math.nan for LastUpdateCPU(s), DiskSpace(MB), MemoryUsed(MB) and LoadAverage whenever a job ends before the first Watchdog cycle (20-30 min). NaN is not valid JSON and cannot be stored in the job parameters backends; when forwarded to diracx it makes the whole metadata update fail (DIRACGrid/diracx#582). - Watchdog: omit usage summary keys when no sample was collected instead of reporting NaN - JobReport: drop non-finite parameter values with a warning so no producer can send them - JobWrapper / dirac-wms-cpu-normalization / Watchdog: validate CPUNormalizationFactor with math.isfinite before using or storing it, as a nan/inf CS correction value survives float parsing silently Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fstagni
reviewed
Aug 7, 2026
| mocker.patch("DIRAC.WorkloadManagementSystem.Client.JobStateUpdateClient", side_effect=MagicMock()) | ||
|
|
||
| jr = JobReport(123) | ||
| res = jr.setJobParameter("LoadAverage", float("nan"), sendFlag=False) |
Contributor
There was a problem hiding this comment.
What if:
Suggested change
| res = jr.setJobParameter("LoadAverage", float("nan"), sendFlag=False) | |
| res = jr.setJobParameter("LoadAverage", math.nan, sendFlag=False) |
?
Comment on lines
142
to
+145
| self.cpuPower = gConfig.getValue("/LocalSite/CPUNormalizationFactor", 1.0) | ||
| if not math.isfinite(self.cpuPower): | ||
| self.log.error("Ignoring non-finite CPUNormalizationFactor from configuration", str(self.cpuPower)) | ||
| self.cpuPower = 1.0 |
Contributor
There was a problem hiding this comment.
Should we take the occasion to change the default of 1.0 ?
I also do not see why it would ever be a non-finite number.
|
|
||
| gLogger.info("Applying a correction on the CPU power:", corr) | ||
| cpuPower = round(db12Result / corr, 1) | ||
| if not math.isfinite(cpuPower): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #6938,
Watchdog.__getUsageSummaryreportsmath.nanforLastUpdateCPU(s),DiskSpace(MB),MemoryUsed(MB)andLoadAveragewhenever the corresponding sampling list is empty. Every job shorter than the first watchdog cycle reports four NaN job parameters.NaN is not valid JSON (RFC 8259): when these parameters are forwarded to diracx, OpenSearch rejects the document and the whole update fails with a 500 (DIRACGrid/diracx#582). With diracx's stricter validation, the batch is rejected with a 422, losing the valid parameters sent alongside.
In this PR, we make sure no such value can go through any JSON sent to diracx.
BEGINRELEASENOTES
*WorkloadManagement
FIX: do not report non-finite (NaN/Infinity) job parameters from the Watchdog and JobWrapper; validate CPUNormalizationFactor
ENDRELEASENOTES