feat(dsql): Add system diagnostics workflow (Workflow 12)#207
Open
Morlej wants to merge 8 commits into
Open
Conversation
Add CloudWatch AAS-based system diagnostics to the DSQL skill. Uses PromQL queries against db.active_sessions.avg to detect temporal anomalies in wait event distribution and identify regressed queries, then routes to Workflow 9 (Query Plan Explainability) for per-query investigation. OTel attribute names use the proposed naming convention: - db.wait.event, db.wait.class, db.session.state - db.query.id, db.query.normalized_text - aws.auroradsql.session.role.arn, application.name New files: - references/system-diagnostics/workflow.md — 6 diagnostic sub-workflows - references/system-diagnostics/wait-events.md — canonical wait event reference - references/system-diagnostics/promql-patterns.md — reusable PromQL templates Also: - Adds cloudwatch MCP server to .mcp.json (disabled by default) - Bumps plugin version to 1.5.0
Add a decision table before Common Workflows that routes performance complaints to Workflow 12 (System Diagnostics) instead of allowing them to fall through to Workflow 9 (Query Plan Explainability) directly. Rule: when in doubt, start with Workflow 12 — it identifies specific queries and routes to Workflow 9 with context.
- Use correct get_promql_label_values syntax with match parameter - Add note that calls without match filter return empty - Add PromQL syntax rules: quote labels with dots/@, use __name__ selector - Add explicit discovery step to Workflow 1 - Fix promql-patterns.md to show actual tool parameter names (label_name, match)
Replace separate numbered workflows (1-6) with a single diagnostic procedure of 5 mandatory phases. The agent MUST execute ALL phases before presenting results — no stopping at the first finding. Phases: 1. Discovery and Baseline Comparison (distribution shifts) 2. Top-SQL Regression Detection (new/growing queries) 3. Workload Attribution (application/role changes) 4. Commit and OCC Analysis (volume vs conflicts) 5. Inflection Point Detection (when did it change) Adds 'Presenting Results' section mandating a unified report across all dimensions before handoff to Workflow 9.
Morlej
requested review from
a team,
krokoko,
scottschreckengaust and
theagenticguy
Morlej
requested review from
Benjscho,
amaksimo,
anwesham-lab,
jaichabria,
pkale,
praba2210 and
spencercorwin
Contributor
|
Thanks @Morlej ! |
Signed-off-by: James Morle <7480305+Morlej@users.noreply.github.com>
Workflow 12 (System Diagnostics) required the CloudWatch MCP server but gave no way to enable it, and agents that hit unavailable PromQL tools had no recovery guidance. Address the observed failure modes: - Add a CloudWatch MCP Server section to mcp/mcp-setup.md: enabling the shipped-disabled server, AWS_REGION/AWS_PROFILE config, the PromQL-enabled region limitation, and the session restart needed for its tools to register (a mid-session enable shows "Connected" but exposes no callable tools until restart). - Add AWS_REGION/AWS_PROFILE placeholders to the disabled cloudwatch entry in .mcp.json so users see what to fill in. - workflow.md: add an explicit "If the PromQL tools are unavailable" callout in Prerequisites (enable + restart, no fabrication/CLI fallback) and a matching Error Handling row, so a blocked agent surfaces the fix rather than only reporting missing data. - Document the discovery-window gotcha: get_promql_label_values defaults to a window ending "now" and returns empty for older data; pass explicit start/end. An empty result means "no data in window," not "no such cluster." - Fix "6 diagnostic workflows/sub-workflows" -> "5 diagnostic phases" in SKILL.md (both the reference table and Workflow 12 section). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves the 1.5.0 version collision with EF Core PR awslabs#208 (both this branch's system-diagnostics work and awslabs#208 independently claimed 1.5.0). Bumps to 1.6.0 across plugin.json, codex plugin.json, and marketplace.json. Adds eval coverage for Workflow 12 (system diagnostics), which previously had none: 3 triggering cases and 1 functional AAS-interpretation eval. Corrects stale eval README counts (37 trigger cases, 15 functional prompts / 54 assertions) and documents evals 13-15. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds CloudWatch AAS-based system diagnostics to the DSQL skill as Workflow 12.
What it does
Uses PromQL queries against
db.active_sessions.avgOTel metrics to run a mandatory full diagnostic sweep across 5 phases:Routes identified queries to Workflow 9 (Query Plan Explainability) for per-query investigation.
OTel attribute naming
Uses the proposed naming convention:
db.wait.event,db.wait.class,db.session.statedb.query.id,db.query.normalized_textaws.auroradsql.session.role.arn,application.nameKey design decisions
matchparameter (documented as critical rule)Files
references/system-diagnostics/workflow.md— 5-phase diagnostic procedurereferences/system-diagnostics/wait-events.md— canonical DSQL wait event referencereferences/system-diagnostics/promql-patterns.md— reusable PromQL templates.mcp.json— adds cloudwatch MCP server (disabled by default)SKILL.md— updated description, tags, reference table, Workflow 12, Performance Routing tableEval coverage
trigger_evals.json— 3 Workflow 12 triggering cases (cluster performing differently, wait-event/AAS breakdown, "where is compute time spent"); now 37 cases (22 trigger / 15 no-trigger)evals.json— functional eval chore(doc): add troubleshooting guide #15: AAS wait-event interpretation (identifies the shifted event vs baseline, rules out proportional load growth, makes no absolute-AAS claim, defers per-query root cause to Workflow 9); now 15 prompts / 54 assertionsVersioning
Bumped
databases-on-awsto 1.6.0 (plugin.json, codex plugin.json, marketplace.json). This resolves a version collision: EF Core PR #208 landed on main and took1.5.0, which this branch had also claimed for Workflow 12. 1.6.0 places the diagnostics feature on top of current main.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the project license.