Skip to content

test: add colocated core component acceptance suite - #841

Open
kvinwang wants to merge 201 commits into
nextfrom
codex/core-components-test-infrastructure
Open

test: add colocated core component acceptance suite#841
kvinwang wants to merge 201 commits into
nextfrom
codex/core-components-test-infrastructure

Conversation

@kvinwang

@kvinwang kvinwang commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR adds the core-component acceptance suite and organizes the complete suite directly under test-suites/ so specifications, automation, fixtures, inventories, manifests, and retained audit evidence have one clear home.

  • Adds the deterministic dstack-test runner, dashboard, registry validation, fixture providers, and component harnesses.
  • Adds 358 acceptance cases covering guest OS, VMM, KMS, gateway, verifier, and cross-component integration behavior.
  • Colocates single-case entrypoints with their case.md specifications as run.py, run.sh, or run.cjs.
  • Keeps multi-case and location-sensitive harnesses in shared/automation/.
  • Distributes metadata across suite, chapter, section, and case directories; the runner discovers cases in directory-name order.
  • Uses each case-local metadata.json as the authoritative case-to-entrypoint binding.
  • Keeps fixture providers and images under shared/fixtures/, inventories under catalog/, environment manifests under manifests/, and retained historical evidence under audit/.
  • Adds suite authoring, execution, reporting, and methodology documentation.
  • Excludes product/runtime changes and source-local unit tests; those changes were split into dedicated PRs.

Layout

test-suites/
├── runner/
├── cases/
├── shared/
│   ├── automation/
│   └── fixtures/
├── catalog/
├── manifests/
└── audit/

Separation invariant

The PR targets next. Its changed paths are limited to:

  • REUSE.toml
  • docs/testing/**
  • test-suites/**

There are no product/runtime or source-local test changes under dstack/** or os/**.

Verification

  • Registry: 357 scripted / 357 promoted / 0 findings (358 total cases)
  • Runner unit tests: 37 passed
  • prek run --all-files: passed
  • reuse lint: passed
  • git diff --check origin/next...HEAD: passed

Copilot AI review requested due to automatic review settings July 31, 2026 02:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@kvinwang kvinwang changed the title test: add core component acceptance infrastructure [STACKED on #840] test: add core component acceptance infrastructure Jul 31, 2026
@kvinwang
kvinwang force-pushed the codex/core-components-test-infrastructure branch from 7a21a97 to 313efb9 Compare July 31, 2026 03:34
@kvinwang kvinwang changed the title [STACKED on #840] test: add core component acceptance infrastructure test: add core component acceptance infrastructure Jul 31, 2026
@kvinwang
kvinwang changed the base branch from codex/core-components-product-fixes to master July 31, 2026 03:34
@kvinwang
kvinwang force-pushed the codex/core-components-test-infrastructure branch 2 times, most recently from 8cac7e2 to 4c99500 Compare August 18, 2026 09:14
@kvinwang
kvinwang force-pushed the codex/core-components-test-infrastructure branch from 4c99500 to dd0f6b8 Compare August 18, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants