[mcp_gateway] Replace MLflow historical data with S3-based KPI pipeline - #121
[mcp_gateway] Replace MLflow historical data with S3-based KPI pipeline#121ashtarkb wants to merge 8 commits into
Conversation
Switch from querying MLflow for historical run comparison to using Caliper's standard S3 import/export pipeline for KPI historical data. - Add S3 import/export configuration to config.yaml (bucket, vault, filters) - Enable KPI CSV generation and historical data analysis - Rewrite notifications to compare against S3-imported historical KPIs instead of querying MLflow API directly - Add version-aware filtering: skip same-version runs when selecting the comparison baseline for Slack notifications - Remove MLflow-specific run name parsing and metric key mapping Co-authored-by: Cursor <cursoragent@cursor.com>
|
Skipping CI for Draft Pull Request. |
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… matching - Add `analyze_kpis` hook to Caliper's PostProcessingPlugin interface, allowing projects to implement custom regression analysis - Update caliper's analyze step to delegate to plugin when implemented - Create mcp_gateway analyze module with full matching logic: - SHA vs semver version type filtering - Load config matching (num_servers + users) - Preset matching - Same-version deduplication - Update notifications to reuse analyze module's matching logic Co-authored-by: Cursor <cursoragent@cursor.com>
openshift-ci
Bot
added
the
needs-rebase
…orical-data Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # fournos/gitops/base/workflows/task-forge-step.yaml
…egation Restore the upstream run_kpi_analysis() code exactly as-is and add only the plugin delegation logic (_load_plugin + try plugin.analyze_kpis first). Co-authored-by: Cursor <cursoragent@cursor.com>
openshift-ci
Bot
removed
the
needs-rebase
🔴 Execution of
|
- Add _load_kpi_file() helper that handles both single-document JSON
and JSONL formats, converting flat JSONL records into the hierarchical
{"tests": [...]} structure that plugins expect.
- Fix UnboundLocalError for sys in commands.py analyse_kpis_cmd by
moving import sys to function scope.
Co-authored-by: Cursor <cursoragent@cursor.com>
🔴 Execution of
|
🟢 Execution of
|
🔴 Execution of
|
🔴 Execution of
|
🔴 Execution of
|
🛑 Execution of
|
🔴 Execution of
|
🔴 Execution of
|
🛑 Execution of
|
🛑 Execution of
|
🔴 Execution of
|
4cacdc6 to
1ec55a8
Compare
🟢 Execution of
|
openshift-ci
Bot
added
the
needs-rebase
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@ashtarkb can this be closed? |
2 participants
Summary
kpis.jsonfiles with version-aware filtering (skips same-version runs)Details
The previous implementation queried MLflow directly for historical run data, which had timing issues (called during
do_testbefore MLflow export) and didn't align with how other projects (skeleton, llm_d) handle historical data.This PR switches to Caliper's standard approach:
kpis.jsonintohistorical_data/analyse_kpisstep compares current vs historical KPIshistorical_data/and filter bymcp_gateway_versionto find the most recent different version for meaningful comparisonTest plan
mcp-gateway-cpt-datais created and accessibleMade with Cursor