docs: add Run 2 section to GA test report#38
Open
shreemaan-abhishek wants to merge 1 commit into
Open
Conversation
📝 WalkthroughWalkthroughThis PR updates the GA test report documentation to record Run 2 test execution results against API7 EE 3.9.12. The addition documents the test environment, regression verification outcomes, four newly discovered issues, and the phase acceptance checklist. ChangesRun 2 Test Results and Acceptance
🎯 1 (Trivial) | ⏱️ ~3 minutes 🚥 Pre-merge checks | ✅ 5 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull request overview
Adds a second execution record (“Run 2”) to the GA smoke-test report, capturing post-merge verification of prior fixes and documenting newly discovered issues against a real API7 EE 3.9.12 instance.
Changes:
- Appends a “Run 2” section with environment details, regression-check results, and exit-criteria status.
- Documents newly observed issues/findings from the second pass (bugs/UX/cosmetic).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| ## Summary | ||
|
|
||
| Re-run after the Run 1 fixes (PR #31) and the `plugin-config` removal (PR #34) merged to master. **All 6 targeted regression checks hold.** Each of the 13 core resources round-trips. **4 new findings surfaced (3 bugs + 1 cosmetic)** — none block GA, but the 3 bugs each warrant a sub-issue and a test-before-fix. |
Comment on lines
87
to
+93
| ## Follow-ups | ||
|
|
||
| - **Task #2** (remove the `plugin-config` standalone command) is still outstanding. Once done, Phase C also expects the declarative `plugin_configs` top-level section to be rejected — the `service_templates` rejection added here is the template for that change. | ||
|
|
||
| --- | ||
|
|
||
| # Run 2 (post-#31, post-#34) |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/ga-test-report.md`:
- Around line 106-129: The summary count is inconsistent with the New findings
table: the summary sentence ("4 new findings surfaced (3 bugs + 1 cosmetic)")
must be reconciled with the table entries (R2-1..R2-5) — either update the
summary to "5 new findings surfaced (3 bugs + 1 UX + 1 cosmetic)" and add R2-2
to the PR description (alongside the existing references to
R2-1/R2-3/R2-4/R2-5), or remove R2-2 from the table, or add a clarifying note
why R2-2 is excluded (e.g., "tracked separately / post-GA candidate"); locate
the summary sentence and the New findings rows (R2-2) in the doc and make the
corresponding change so the numeric totals and PR description (`#35/`#36/#37
mapping) are consistent.
- Line 139: The markdown table row missing a closing pipe should be fixed by
appending a trailing '|' to the row string "Each new bug gets a tracked
sub-issue with test-before-fix protocol | ⏳ pending sub-issue creation" so the
row becomes properly terminated with a final pipe character to satisfy the MD055
table-pipe-style rule.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: d21dbcd5-44d5-490f-b00e-6896e95db36b
📒 Files selected for processing (1)
docs/ga-test-report.md
Comment on lines
+106
to
+129
| ## Summary | ||
|
|
||
| Re-run after the Run 1 fixes (PR #31) and the `plugin-config` removal (PR #34) merged to master. **All 6 targeted regression checks hold.** Each of the 13 core resources round-trips. **4 new findings surfaced (3 bugs + 1 cosmetic)** — none block GA, but the 3 bugs each warrant a sub-issue and a test-before-fix. | ||
|
|
||
| ## Regression checks (all pass) | ||
|
|
||
| | Check | Source | Result | | ||
| |---|---|---| | ||
| | `ssl update` without `--cert/--key` → clean error | PR #31 BUG-1 | ✅ | | ||
| | `plugin-metadata get -o yaml` → YAML map | PR #31 BUG-2 | ✅ | | ||
| | `plugin get -o yaml` → YAML (not JSON) | PR #31 BUG-3 | ✅ | | ||
| | `stream-route create` without `--name` → error; `-f -o yaml` → YAML map | PR #31 BUG-4 + review fixup | ✅ | | ||
| | `config validate` rejects all 4 unsupported sections (incl. empty `[]`) | PR #31 BUG-5 + PR #34 | ✅ | | ||
| | `a7 plugin-config` / `upstream` / `consumer-group` / `service-template` → `unknown command` | PR #34 | ✅ | | ||
|
|
||
| ## New findings | ||
|
|
||
| | # | Severity | Resource | Finding | Disposition | | ||
| |---|---|---|---|---| | ||
| | R2-1 | 🟡 Bug | route | README documents `a7 route update <id> --desc "..."` but neither `route create` nor `route update` exposes a `--desc` flag. Description is only settable through `-f`. | Sub-issue + E2E | | ||
| | R2-2 | 🟡 UX | route | `a7 route list -g default` errors with `--service-id is required by API7 EE`. The e2e helper iterates services to aggregate routes; the CLI doesn't. | Sub-issue (post-GA candidate) | | ||
| | R2-3 | 🟡 Bug | credential | `a7 credential create smoke-cred-X --consumer Y ...` returned a server-assigned UUID, ignoring the positional id. (Run 1 noted this as "intended" via `TestCredential_CreateWithPositionalID` — needs re-confirmation; if intended, drop the misleading `[id]` from the help.) | Sub-issue + decision | | ||
| | R2-4 | 🟡 Bug | global-rule | `a7 global-rule create --id X --plugins-json '{"cors":...}'` ignores `--id`; resource is created at `id=cors`. Run 1 noted "value is ignored by EE" as a minor; treat as a real bug: the CLI shouldn't accept a flag it will silently override. | Sub-issue + E2E | | ||
| | R2-5 | 🟢 Cosmetic | stream-route | `-o yaml` renders `created_at: 1.779521636e+09` in scientific notation. Should be plain integer. | Note only | |
There was a problem hiding this comment.
Inconsistency between summary count and findings table.
Line 108 states "4 new findings surfaced (3 bugs + 1 cosmetic)" but the New findings table (lines 121-129) contains 5 entries:
- R2-1, R2-3, R2-4: 3 bugs ✓
- R2-5: 1 cosmetic ✓
- R2-2: 1 UX issue (not accounted for in the summary)
Additionally, the PR description lists four findings (#35, #36, #37, + cosmetic) which correspond to R2-1, R2-3, R2-4, and R2-5, but does not mention R2-2 (route list requiring service-id).
Please either:
- Update the summary to "5 new findings surfaced (3 bugs + 1 UX + 1 cosmetic)" and add R2-2 to the PR description, or
- Remove R2-2 from the table if it's not considered a new finding for this run, or
- Clarify why R2-2 is excluded from the count (e.g., if "post-GA candidate" disposition means it's tracked separately).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/ga-test-report.md` around lines 106 - 129, The summary count is
inconsistent with the New findings table: the summary sentence ("4 new findings
surfaced (3 bugs + 1 cosmetic)") must be reconciled with the table entries
(R2-1..R2-5) — either update the summary to "5 new findings surfaced (3 bugs + 1
UX + 1 cosmetic)" and add R2-2 to the PR description (alongside the existing
references to R2-1/R2-3/R2-4/R2-5), or remove R2-2 from the table, or add a
clarifying note why R2-2 is excluded (e.g., "tracked separately / post-GA
candidate"); locate the summary sentence and the New findings rows (R2-2) in the
doc and make the corresponding change so the numeric totals and PR description
(`#35/`#36/#37 mapping) are consistent.
| | All Run 1 fixes hold against current master | ✅ 6/6 regression checks pass | | ||
| | Phase C unsupported commands gone (including `plugin-config`) | ✅ all 4 return `unknown command` | | ||
| | Phase D `config dump/validate` work on a clean instance | ✅ (no Run 2 dirty-state diff test, but Run 1 covered diff/sync convergence) | | ||
| | Each new bug gets a tracked sub-issue with test-before-fix protocol | ⏳ pending sub-issue creation |
There was a problem hiding this comment.
Fix table formatting: missing trailing pipe.
Line 139 is missing the trailing | character to properly close the markdown table row.
📝 Proposed fix
-| Each new bug gets a tracked sub-issue with test-before-fix protocol | ⏳ pending sub-issue creation
+| Each new bug gets a tracked sub-issue with test-before-fix protocol | ⏳ pending sub-issue creation |As per coding guidelines, the static analysis tool flagged this as MD055 table-pipe-style violation.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| | Each new bug gets a tracked sub-issue with test-before-fix protocol | ⏳ pending sub-issue creation | |
| | Each new bug gets a tracked sub-issue with test-before-fix protocol | ⏳ pending sub-issue creation | |
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 139-139: Table pipe style
Expected: leading_and_trailing; Actual: leading_only; Missing trailing pipe
(MD055, table-pipe-style)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/ga-test-report.md` at line 139, The markdown table row missing a closing
pipe should be fixed by appending a trailing '|' to the row string "Each new bug
gets a tracked sub-issue with test-before-fix protocol | ⏳ pending sub-issue
creation" so the row becomes properly terminated with a final pipe character to
satisfy the MD055 table-pipe-style rule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the second pass of
docs/ga-test-plan.mdagainst master @ac31b9d(post-PR #31, post-PR #34) on a real API7 EE 3.9.12 instance.routemissing--descflagcredentialpositional id behaviorglobal-ruleignores--id-o yamlCloses #25. Part of #22.
Summary by CodeRabbit