Repository navigation
Merge integration/all-specs into main (142 specs, Jul–Oct 2026) - #74
Merged
Merged
Conversation
Live acceptance (JP6 fleet job ea52801f) surfaced that the SSM agent
environment on the build server configures no default AWS region, so the
prune script's region-less 'aws ecr describe-images' exited 255 ('You
must specify a region') and every stale per-version candidate was
retained as ECR_UNVERIFIABLE (fail-open worked as designed, but the
~74 GB of confirmed-pushed 1.0.6 generations stayed, leaving 49 GB free
< the 60 GB threshold and failing the build fast with RUNNER_DISK_FULL).
Derive the region from the ECR registry hostname (the candidate pattern
guarantees *.dkr.ecr.<region>.amazonaws.com/dda/*) and pass it
explicitly. Verified host-side: bash -n clean;
test/backend-test/build_server_disk_pruning 43 passed (the stub-argv
PBTs and the read-only-transcript guard accept the appended --region);
script unpinned by any golden, no rebaseline.
GET /datasets scanned the literal S3 prefix 'None/' for every use case since the single-account-setup change hardcoded None in the prefix slot of get_data_bucket_and_credentials, silently emptying the dataset dropdowns in Synthetic Data, Dataset Browser, and training creation. Derive the prefix from data_s3_prefix/s3_prefix with '' coercion, add defensive None-safety in list_datasets, and return base_prefix as '' instead of null. Exploration + preservation property tests included (26 tests, run standalone).
…-pruning complete, session evidence)
…role Single-account synthetic generation failed with AccessDenied on s3:GetObject: the SyntheticDataHandlerRole had no S3 statements, so the worker's execution-role fallback could not read source images or write previews/manifests. Replicate the compute stack's reviewed data-plane grant pattern (bucket-level ListBucket/GetBucketLocation/GetBucketTagging plus object-level GetObject/PutObject, dataBucketAllowlist-scopable, no control-plane actions, no aws:ResourceTag condition) onto the handler role and pass the allowlist through from app.ts. CDK assertions tests cover the new grants, allowlist normalization, existing-statement preservation, and a universal no-control-plane check.
…ycle-aware availability Nova Canvas is LEGACY-locked in us-east-1 (InvokeModel rejects with a ResourceNotFoundException for accounts without 30-day usage), leaving the synthetic data pipeline with zero usable models. Add stability.stable-image-inpaint-v1:0 (invoked via the us.stability.stable-image-inpaint-v1:0 inference profile, verified invokable on the account) behind a per-provider request adapter: deterministic seed-derived mask rectangles rendered as binary PNG masks, mask_region recorded for auto-annotation, Amazon request bodies byte-preserved against a frozen oracle. The availability filter is now lifecycle-aware, hiding LEGACY models from the dropdown while keeping their catalog entries. Per-task failures distinguish access denials from legacy lifecycle lockouts. No frontend or IAM changes.
The preview endpoint hard-capped results at 20 presigned URLs with no pagination; lexicographic S3 ordering made every normal-*.jpg in a 63-image dataset unreachable, blocking the inpainting path. Add offset/limit pagination to get_image_preview (all keys collected, page sliced, presigned per page, true total_found plus offset/limit/has_more, per-page cap 50, legacy defaults preserved), a typed offset surface in the API client, and a paged 6x2 thumbnail grid in the wizard with Previous/Next controls, a showing-A-B-of-TOTAL indicator, and selection that persists across pages into submission. Updated the dataset-discovery preservation test for the by-design total_found semantics change (truncated count -> true total).
…ful status, per-arch packaging
Spec: .kiro/specs/vllm-model-reload-after-backend-restart
Incident: qwen3-vl-8b-instruct was silently dropped by backend restarts
(awscrt-abort kills) and stuck 'loading' forever - the loaded engine lives in
the backend process and nothing ever re-issued the load for the still-staged
model (gpuActiveModels: 0, models: {}; every generate 409; feature-config
LOADING indefinitely until a human restarted the model component).
Device leg:
- NEW src/backend/vllm_runtime/reconciler.py: VllmReconciler daemon thread
started by start_vllm_runtime(); one STAGED snapshot at backend start, then
a sequential re-drive of each load through the same loopback model-control
endpoint the component Startup uses (never manager.load() directly);
bounded (30, 120, 480) retry schedule with the KV-OOM single unload->reload
recovery per attempt; exhaustion -> FAILED with retained reason, one
prominent ERROR, never a retry storm; per-model isolation; vLLM-free images
(JP5/x86) byte-identical.
- manager.py + constants.py: Unload_Tombstone (.dda_explicit_unload) - an
explicit unload suppresses reconciliation until an explicit load or the
component Startup's atomic re-stage re-arms it; reporting-only
ModelState.UNLOADED for staged-but-tombstoned repos.
- feature_configs_utils.py + text_generation.py: truthful status surfaces -
UNLOADED->STOPPED / "unloaded" 409 category (fail fast instead of burning
the workflow binding's 240 s budget); STAGED->LOADING retained because the
reconciler now makes it true by construction.
Portal leg (defect 1.6):
- workflow_packaging.py: resolve_model_components drops the vLLM singular-map
short-circuit that emitted the unsuffixed base name (the incident's exact
HARD dep model-vllm-qwen3-vl-8b-instruct, which dragged LocalServer
.arm64JP6 1.0.59 onto the JP7 Thor); vLLM references now resolve
per-architecture through the record's platform-suffixed Per_JetPack
evidence (fail closed on uncovered archs; the base name is never emitted).
Plus the defense-in-depth arch-contradiction guard: refuse packaging when a
resolved model component's recipe HARD-depends on a LocalServer variant
serving none of the selected architectures (warn-and-proceed on read
failures).
Task-6 gate verification (all green, host-side venv + moto):
- security guard pair 4 passed, 3 skipped (no preservation-tracked file
touched, no baseline rebaselined - verified, not assumed)
- full device spec suite test/backend-test/vllm_model_reload = 45 passed
- pinned device suites at task-2 baselines, unmodified: vllm_runtime 28,
vllm_runtime_tests 1, text_generation 15, deploy_reliability 72,
workflow_engine LLM-binding 42
- portal suites WITH conftest: arch-dependency trio 30; multi-arch publish
7/12/23/3; vision packaging 3/9/4; dependencies exploration 2; localserver
preservation 10; vllm packaging preservation 1; repointed
vllm_resolution_preservation 9; recipe preservation 5; llm model-name
preservation 7
Conscious pinned-suite repoints (recorded first on the unfixed tree, then
repointed - never weakened or deleted; requirement 2.6 supersedes the old
singular-resolution contract for vLLM-shape records):
- test_workflow_packaging_vllm_resolution_preservation.py: the two recorded
old-contract legs - singular-map verbatim resolution -> legacy records fail
closed per-architecture; base-name dependency emission golden ->
suffixed-only emission (base name never appears)
- user-approved extension to the same forbidden contract fixture-embedded in
four more suites: test_vision_model_packaging_preservation.py (vllm-shape
leg -> suffixed-only), test_workflow_packaging_dependencies_exploration.py
(both tests, modern record shapes), test_workflow_packaging_recipe_
preservation.py (fixture-only), test_property_llm_model_name_
preservation.py
- plus the task-3.2 domain-enforcement fix in the manager identity PBT
harness (unlink-after-unload keeps the property in its declared
no-tombstone domain)
Remains hardware-gated (tasks 9-14): sequential component builds JP7-first
per builds.md, portal deploy strictly after builds, Session A on jetson-thor1
(fresh-deploy single-load, restart reload, churn convergence, tombstone,
terminal-failure truthfulness, sustained health), JP6 follow-on smoke, JP5/
x86 inertness smoke, post-deploy portal re-package + closure verification.
…ine.json The reconciler wiring (vllm-model-reload-after-backend-restart task 3.4, commit 4d22725) intentionally edited src/backend/app.py. The security preservation gate's sibling-spec guard (test_preservation_iam_out_of_scope_guard.py::test_sibling_spec_files_unchanged) pins that file's sha256 in iam_out_of_scope_baseline.json; the pin was missed because the host-side pre-push gate runs only the out-of-scope guard pair, while the full in-container gate runs this suite too. Fleet JP7 job be6f8c00 failed on exactly this assertion. Conscious rebaseline per builds.md: d697bafe... -> d61f996a... (the committed 4d22725 app.py). Full preservation suite host-side: 138 passed, 8 skipped.
Deployment revisions shipped aws.greengrass.ShadowManager bare (UI revise preload resubmits it; workflow revisions copy it verbatim), permanently disarming the synchronize auto-include after a target's first revision -- dda-model-status never reached revised devices fleet-wide. Fix: shared ensure_shadow_manager_sync helper in deployments.py (absent -> fresh-add unchanged; bare/corrupt -> full portal merge injected; stale -> order-preserving union of namedShadows; compliant -> byte-identical no-op) applied at create_deployment and presence-gated at create_workflow_deployment; frontend adds ShadowManager to the preload autoManaged skip set. ONE conscious pinned-test repoint (recorded verbatim in the preservation suite + a comment): test_caller_supplied_shadow_manager_is_not_overridden -> test_caller_supplied_shadow_manager_keeps_version_and_gains_portal_merge. Verified host-side: spec suites 20 (3 exploration + 5 preservation + 6 properties + 6 units), test_deployment_shadow_manager 4 + test_model_status_shadow_sync 2, workflow-deploy suites 38 at baseline, frontend preloadShadowManager 4 + archFilter 11, full vitest 1305 green, npm run build clean, security guard trio green (no preservation-tracked file touched). Device leg (thor1 revision-11) pending post-deploy.
…ev-11 + reconciler session A; JP6 1.0.61 blocker) - shadowmanager-sync-config-on-revision task 9: rev-11 (abf80fc1) bare-entry completion with the full three-shadow merge verified live; dda-model-status cloud shadow materialized and syncing; portal DevicesHandler serves model_status; workflow pass-through verified twice (thor1 rev 13, jp622 rev 73 = second-device spot-check). - model-gpu-fallback-visibility task 11: shadow/portal leg PROVEN end-to-end (cloud shadow + portal data path); stays [~] pending the fallback-window (degraded) leg. - vllm-model-reload-after-backend-restart: task 11 steps 1-6 PASSED on thor1 1.0.8 (single-load, restart/churn convergence, tombstone, terminal-failure truthfulness, ~33 min sustained health; step 4d --cleanup leg remains); task 14 deployment leg DONE (workflow 11.0.0 -> thor1 rev 13, zero JP6 lineage) - marked [x]; task 12 BLOCKER recorded (jp622 rejected 1.0.61, FAILED_ROLLBACK_COMPLETE, two hypotheses, rev-73 preload warning); task 13 blocked on device access.
…k (dataset discovery, S3 perms, Stability models, picker pagination)
Follow-up to the spec/source-image-picker-pagination merge. When another test file in the same pytest run activates conftest's session-scoped aws_stack mock, the nested per-example mock_aws() contexts in the image preview pagination and dataset discovery preservation suites share that outer moto backend and no longer wipe S3 state on exit, so objects from earlier hypothesis examples leaked into later ones (order-dependent failures; reproduced unchanged on the source branch, so pre-existing there, not a merge regression). _seed_bucket now purges the bucket before seeding so every example is hermetic. No assertions were changed.
…ployment race on JP6, not an image regression)
…ls + grant UserAdmin the audit-finalize Query
Spec: .kiro/specs/user-manager-datalabeler-role (bugfix; Incident Record
2026-08-17, account 164152369890).
Defect 1 - the User Manager modals never offered DataLabeler. The frontend
UserManagerModals.tsx kept its own five-role PORTAL_ROLES while the backend
user_admin.py already accepted six, and both the create-user and change-role
dropdowns map that same array. No labeler account could be created or
converted through the UI, so LabelingTeams add-member candidates
(role === 'DataLabeler') were permanently empty and dda-data-labeling was dead
end-to-end. Fix: 'DataLabeler' appended LAST to PORTAL_ROLES (one string; the
five keep their order, matching the backend tuple) plus the doc comment.
Defect 2 - the UserAdmin Lambda role lacked dynamodb:Query on
dda-portal-audit-log. finalize_audit_event recovers the (event_id, timestamp)
range key with a Query, but createLambdaRole('UserAdmin') only received
auditLogTable.grantWriteData. Because every user-admin mutation follows the
audit-before-effect protocol, EVERY mutation 500s AFTER its Cognito effect was
applied and its audit entry stuck at 'pending'. Verified live:
AccessDeniedException on UserAdminRole2557D264 for dynamodb:Query against
table/dda-portal-audit-log, request id 79dc2dd8-2e10-486a-a0e6-aec5ef805d37
(2026-08-17 17:03Z remediation create - the account was created, the caller
still got "Internal server error"). Fix: one scoped grant in compute-stack.ts,
props.auditLogTable.grant(userAdminHandler, 'dynamodb:Query') - narrower than
grantReadData (no Scan/GetItem) and scoped to this handler only, leaving the
base createLambdaRole write-only. No backend code change; finalize keeps its
Query call site and raise-on-failure contract, so auditing is never weakened.
ONE conscious pinned-test repoint (design Decision 2), never weakened or
deleted: UserManagerModals.test.tsx 'offers exactly the five defined roles with
the current role preselected (Requirement 5.2)' -> the six-role contract;
dda-data-labeling Req 2.1 (DataLabeler assigned/revoked through the existing
user administration functions) supersedes the portal-user-manager 5.2 count.
The old name and assertion are recorded verbatim in an adjacent comment and in
the spec's task-2 OUTCOME, so the diff is auditable. No other test in that file
was touched.
Host-side verification (task 5, all gates green):
- frontend: 81 tests across the 5 src/pages/admin suites (UserManagerModals 28
incl. the repoint, UserManager 21 unmodified, UserManagerSyncPanel 13
unmodified, new dataLabelerRole 13, FleetPage 6) + npm run build clean
- infrastructure: 45 tests across the 5 ComputeStack-synthesizing jest suites
(user-admin-audit-grant 9, camera-registry-infra 13, quick-setup-infra 12,
camera-shadow-sync-provisioning 8, workflow-manager-gaps-infra 3) - the new
grant disturbs no existing assertion
- backend: 249 passed at the recorded per-suite baselines (test_user_admin_*
+ test_dda_labeling_rbac_role) plus 2 in the new finalize-path moto test
- security preservation guard pair: 4 passed, 3 skipped - no
preservation-tracked file touched, no baseline rebaselines needed
New tests: UserManagerModals.dataLabelerRole.test.tsx (option lists, six-role
parity, DataLabeler submission shapes), user-admin-audit-grant.test.ts (CDK
static assertions: the Query grant, its scope, the role's pre-existing
statements), test_user_admin_audit_finalize_preservation.py (moto: pending
before the effect, exactly one Query during finalize, terminal result).
Deploy-gated, NOT yet verified in the live account (design Decision 5 - moto
does not enforce IAM, so the deployed-IAM truth is account-tier only): task 7
is the portal deploy (infrastructure + frontend, never while a component build
runs per builds.md, cdk.out moved aside afterwards) and task 8 is the live
verification in account 164152369890 (UI-created DataLabeler account returning
200 with a terminal audit entry, the convert-role path, LabelingTeams gaining a
candidate, and remediation of the stuck-'pending' 2026-08-17 entry).
…he three-branch push verification in the task-6 OUTCOME (doc only)
…ommit 3a9c8e3 in the task-6 OUTCOME
…ification OUTCOMEs (doc only) Task 7: portal deploy succeeded (infra 20:36-20:44Z, frontend 20:44-20:46Z, no rollbacks); live UserAdminRole2557D264 now carries dynamodb:Query on dda-portal-audit-log (+ /index/*); fresh bundle index-CBwQHl-3.js serves the six-role array; cdk.out moved aside, guard pair 4 passed 3 skipped. Task 8: verified against the deployed stack — DataLabeler create returns 201 (was 500) with a terminal 'success' audit entry; convert-role 200 both ways (demo account restored to Viewer) with terminal entries; 3 DataLabeler accounts now available as LabelingTeams add-member candidates. Four pre-fix entries remain 'pending' (incl. three from 2026-07-14 — evidence the IAM gap had been breaking every user-admin mutation since then); reported, not altered.
…urrent-write regression; both have recorded OUTCOMEs)
…arate 1.0.61 KV-OOM regression found) + thor1 cold-first-generate finding
…k in progress)
Commits the in-flight spec for the JP6 vLLM KV-cache OOM regression found
while verifying the vllm-model-reload JP6 leg on LocalServer.arm64JP6 1.0.61,
plus the exploration test suites written against the unfixed tree.
Spec (.kiro/specs/jp6-vllm-kv-cache-oom-regression/):
- bugfix.md: incident record, bug condition, defect/expected/unchanged clauses
- design.md: decisions, correctness properties, fix implementation files,
rollout shape (Leg A portal deploy, then a SINGLE JP6 component build)
- tasks.md: exploration -> preservation -> fix -> fix-check -> gates ->
USER ACTION waves; tasks 0-1 complete, task 2 (preservation) in progress
Exploration suites (expected to FAIL on the unfixed tree by design):
- test/backend-test/jp6_vllm_kv_cache_oom/{fakes.py,test_exploration_jp6_kv_cache_oom.py}
- edge-cv-portal/backend/tests/test_jp6_kv_fit_check_exploration.py
All three compile clean (py_compile).
Also: csi-nvargus-optional task 9 bookkeeping — records that ALL
non-destructive mitigation legs PASSED live on JP7 1.0.7 (capture service
disabled+inactive across a redeploy, watchdog live and silent on a healthy
journal, zero Error(89)); the deliberate degraded reproduction, watchdog
recovery/rate-limit legs, and provisioning verification remain open.
No source/infrastructure file touched; no preservation-tracked file touched;
no component build or portal deploy triggered by this commit. Deploy logs,
cdk.out.bak dirs, driver scripts, CLAUDE.md and hooks deliberately left
uncommitted per the repo's do-not-commit list.
…nes (resumed after a stalled dispatch) Task 2 was marked in-progress but its execution had died ~75 min earlier having produced none of its three deliverables (no pytest running, no OUTCOME block, no files). Resumed from zero and completed. Three new suites, all PASSING on the UNFIXED tree (verified independently before commit: 44 passed, 11 skipped): - edge-cv-portal/backend/tests/test_property_jp6_fit_preservation.py (portal half, WITH conftest): 19 passed, 5 skipped; also green under HYPOTHESIS_PROFILE=ci; reuses the sibling generators verbatim; no hardcoded max_examples - test/backend-test/jp6_vllm_kv_cache_oom/test_preservation_jp6_kv_cache_oom.py: 18 passed, 5 skipped - test/backend-test/jp6_vllm_kv_cache_oom/test_property_jp6_device_preservation.py: 7 passed, 1 skipped The 11 skips are deliberate and reported: 5 fixed-shape legs that bind at task 3.9, and 6 [HARDWARE] legs (H6 thor1, H2 the JP6 ONNX co-tenants, 3.11 device health) declared as skipped tests that RAISE if ever executed host-side, so they cannot be silently faked. Recorded in the task-2 OUTCOME: the S1-S9 sibling originals VERBATIM at their exact files/lines (so every repoint diff in tasks 3.1-3.3/3.7 is auditable), every gate baseline count, and the pinned source hashes (reconciler.py 2bb6f6d3… asserted by a test — this spec may not touch it). Deviations recorded, code untouched: - single-image multi_modal_data["image"] is a BARE PIL image, not a one-element list (list form only for the two-image reference prompt) - the model.json byte-identity pin is scoped to the five pre-existing keys + model, because task 3.1 adds limit_mm_per_prompt additively - test/backend-test/dda_triton cannot be baselined with --noconftest (9/20 prep tests need the repo conftest's caplog; the whole dir does not collect host-side: ModuleNotFoundError: No module named 'google'). Task 5 must run the two prep files WITH conftest; the container run stays authoritative - the frontend touched suites are .tsx, so task 3.4's planned ModelDetail.fitCheckTerms.test.ts should be .tsx No production code touched (git diff over src/ and functions/ is empty), no part of task 3's fix implemented, no component build or portal deploy started, and the serving JP6 device was never contacted (3.11 intact).
…s) + S1-S8 sibling repoints
Spec: .kiro/specs/jp6-vllm-kv-cache-oom-regression (tasks 0-6 complete; this
commit is task 7). 27 paths: Files 1-7 sources, the distributed File 8 tests,
the S1-S8 repoints, the recorded Collision B repoint, and this spec's tasks.md.
Portal Leg A (ships by portal deploy, no component build) - Files 1-3, 7:
- File 1 edge-cv-portal/backend/functions/vllm_fit_check.py: sound Fit_Check -
activation allowance (ACTIVATION_WEIGHT_FRACTION=0.75, labelled an estimate)
+ co-tenancy Fraction_Cap; fits = A AND B; additive FitFinding fields;
Decision 3 message menu (demand-reducing remediation first, raising the
fraction offered only below the cap, never-lower kept).
- File 2 edge-cv-portal/backend/functions/greengrass_publish.py: per-arch
publish gate - ANY failing target arch refuses with 422 (was all-arch);
audited skip_fit_check override preserved.
- File 3 edge-cv-portal/backend/functions/model_import.py: authored
limit_mm_per_prompt engine-config field (default {"image": 1}), reaches
model.json verbatim.
- File 7 edge-cv-portal/frontend/src/services/api.ts: optional types only
(ModelDetail.tsx untouched - types-only outcome of task 3.4).
Device Leg B (ships by ONE aws.edgeml.dda.LocalServer.arm64JP6 build) -
Files 4-6:
- File 4 src/backend/vllm_runtime/manager.py: REMOVED the unbudgeted
limit_mm_per_prompt={"image": 2} setdefault - THE single service-restoring
edit (the published model.json omits the key, so demand returns to 1.0.59's
with no re-package/re-publish); memory_budget preflight call site;
Starvation_Latch (set iff available_after < available_before - tolerance,
cleared by unload); failure classifier (stable category tokens, reason
verbatim); thin-margin WARNING (KV floor / concurrency); two-image request
guard naming the limit.
- File 5 NEW src/backend/vllm_runtime/memory_budget.py: /proc/meminfo
preflight (injected reader; imports no torch/CUDA/vLLM - the parent-process
CUDA-init invariant holds); refuses a doomed load in seconds, not ~4 min.
- File 6 src/backend/dda_triton/vllm_model_prep.py: Decision 3 menu;
LOAD_PREFLIGHT_REFUSED classified before KV_CACHE_HINT_MARKERS; exit 0 on
preflight refusal (deliberate refusal is not a retryable failure).
Sibling repoints, each conscious and recorded (vllm-sizing-and-packaging-errors):
- S5 tests/test_property_fit_check_decision.py - revised by Decision 2:
required now includes the activation allowance, fits is A AND B, message
assertions repointed to the Decision 3 menu (old unconditional "raise
gpu_memory_utilization" assertion replaced by a strictly stronger set);
the negative never-lower assertion KEPT VERBATIM.
- S6 tests/test_vllm_publish_fit_gate.py - revised by Decision 2: all-arch
422 case kept, any-arch gate cases added.
- S7 tests/test_property_engine_config_update_roundtrip.py +
test_property_engine_config_invalid_updates.py - KNOWN_ENGINE_KEYS +=
limit_mm_per_prompt (File 3's authored field).
- S8 tests/test_vllm_engine_config_detail_and_audit.py - expected literals
gain limit_mm_per_prompt {"image": 1}.
- S1-S4 .kiro/specs/vllm-sizing-and-packaging-errors/requirements.md -
requirement text R3.1/R3.6/R3.8/R3.9/R4.2 amended to cite this spec
(REVISED/NARROWED annotations); no other requirement moved.
- S9 (test_property_fit_unverified_never_blocks.py,
test_vllm_fit_check_estimation.py) UNTOUCHED - zero diff, green at their
pre-fix counts.
- Collision B: test/backend-test/vllm_runtime/test_vllm_reference_image_units.py
repointed (recorded in task 3.6's OUTCOME; two-test conscious repoint owned
by vlm-anomaly-reference-parity task 4).
Carried items (deliberately left red for their owners - the same intended
vllm_model_prep.py hash drift 9f3b148a... -> 1d18aaf5...):
- vllm-model-reload-after-backend-restart: PREP_SHA256 pin.
- vllm-hf-cache-persistence: code_baseline.sha256.golden.json entry.
No preservation-tracked file touched; no security-baseline rebaselines needed
(VERIFIED in task 5, not assumed).
UNVERIFIED ON HARDWARE - H1-H8 all still open: H1 published model loads and
serves on JP6; H2 the three ONNX co-tenants keep serving on GPU unchanged;
H3 preflight refusal fast + runtime healthy on device; H4 starvation cascade
behavior; H5 thin-margin/classifier surfaces on device; H6 JP7/thor1
unaffected; H7 NVML-assert root-cause determination; H8 activation-allowance
calibration. Per .kiro/steering/builds.md, this on-device change is NOT "done"
until task 11 passes on ryanorinagxdevkithomelabjp622.
This push triggers no component build and no portal deploy.
…dality keys and size the activation allowance from total multimodal units (portal leg only)
Spec: .kiro/specs/jp6-vllm-kv-cache-oom-regression (task 15; forced by task 11's
ninth OUTCOME block).
WHAT CHANGED (portal leg only — nothing under src/backend/**, no component
build, no device change):
* Widened schema (model_import.py, File 3). `limit_mm_per_prompt` now accepts an
object whose keys are a NON-EMPTY SUBSET of {image, video}: `image` an integer
1..8 (unchanged), `video` an integer 0..8, with a per-sub-key reason quoting
that sub-key's range. Every other sub-key stays fail-closed and `{}` is still
rejected. `ENGINE_DEFAULTS['limit_mm_per_prompt']` is now
{"image": 1, "video": 0}, and the engine-spec endpoint advertises
accepted_keys == ["image", "video"] with both ranges and the "video": 0
guidance (both forms are schema-driven off it).
* WHY — the measured activation halving, 4.93 -> 2.47 GiB. On
ryanorinagxdevkithomelabjp622 (LocalServer.arm64JP6 1.0.62, MemTotal 29.96
GiB), same model, both runs at gpu_memory_utilization = 0.55, verbatim from
vLLM's own profiling:
{"image": 1, "video": 0} -> weights 6.59GiB; non_torch_memory 0.98GiB;
activation peak 2.47GiB; KV Cache 6.43GiB; Maximum concurrency for 4096
tokens per request: 29.41x — READY at 2026-08-19T00:29:52Z
{"image": 1} (video UNBOUND) -> weights 6.59GiB; non_torch_memory 4.76GiB;
activation peak 4.93GiB; KV Cache 0.20GiB; 0.89x — FAILED,
kv-cache-exhaustion (max seq len 4096 > 3664 storable tokens)
Mechanism, from vLLM's own warning: worst-case total multimodal tokens 32768,
of which {'image': 16384, 'video': 16384} are reserved — half the worst case
is video this product never sends. The configuration measured to serve was
UNAUTHORABLE before this change. Honest caveat: non_torch_memory also differed
between the two runs (0.98 vs 4.76 GiB) and swings independently, so the KV
figure and the READY/FAILED outcome are not wholly attributable to the bound;
the activation halving is.
* Units-based activation allowance (vllm_fit_check.py, File 1). New
DEFAULT_VIDEOS_PER_PROMPT = 1, MULTIMODAL_MODALITY_KEYS,
DEFAULT_MULTIMODAL_UNITS = 2, videos_per_prompt(), video_is_authored(),
multimodal_units(); additive FitFinding fields videos_per_prompt /
multimodal_units; activation_allowance(weights, multimodal_units). An
UNAUTHORED `video` key is priced at vLLM's own per-modality default of 1, i.e.
as a SECOND multimodal unit — strictly more expensive than an explicit 0. That
asymmetry is the point. Every message that quotes the allowance still labels it
an ESTIMATE and now names the units it assumed.
* ACTIVATION_WEIGHT_FRACTION is now KNOWN to be ~2x high. At 6.59 GiB of weights
the portal predicts 4.94 / 9.89 GiB against the measured 2.47 / 4.93 GiB —
right ratio (the 2:1 per-unit ratio MULTIMODAL_IMAGE_INCREMENT = 1.0 encodes is
CONFIRMED within 0.01 GiB), magnitude ~2x high (implied ~=0.375 per unit).
RECALIBRATION IS DEFERRED to task 14 / H8 and NOT applied here: the constant is
mirrored in src/backend/vllm_runtime/memory_budget.py and pinned by the
Property 8 parity test, so both legs must move in ONE change and the device leg
needs a JP6 component build. No coefficient was invented to close the gap.
* Property 8 parity REPOINTED, not weakened. The device mirror still counts
images only (device-side, untouched). The suite now pins EXACT parity wherever
the two modules see the same unit count, and the SAFE direction where they do
not: portal_units > device_units, portal required_bytes >= device
required_bytes, portal-accepted implies device-accepted. One-directional
divergence verified over a 99-point grid, 0 counterexamples.
* Sibling repoints (S5/S7/S8), updated never weakened, none deleted or skipped.
S9 (test_property_fit_unverified_never_blocks.py,
test_vllm_fit_check_estimation.py) UNTOUCHED and passing.
* New tests: tests/test_jp6_video_bound_units.py,
tests/test_property_jp6_fit_check_video_units.py.
NOT FIXED BY THIS COMMIT — the four device-leg defects found on hardware
tonight. Each needs a further aws.edgeml.dda.LocalServer.arm64JP6 component
build and is recorded as owed work, not as a verified fix:
(a) the device preflight omits non_torch_memory from `required`, so `required`
is systematically too low (measured ~2 GiB; the term has swung -0.05 ->
8.29 GiB on this device) and its "you need at least util = X" advice is
wrong in the permissive direction;
(b) the Starvation_Latch samples memory BEFORE the driver's asynchronous
release — it reported 9.45 GiB unreturned when ~5.2 GiB came back unaided
with no restart, so it latches on a transient;
(c) an explicit unload SETS the starvation latch it claims to clear, blocking
all further loads for the backend life — the message is currently untrue;
(d) the thin-margin WARNING did NOT fire on a load that reached READY with
0.77 GiB of KV against the 1 GiB floor (the exact H5 shape) — kv_bytes
introspection returned None and the fallback is an invisible debug line.
GATES (from edge-cv-portal/backend, venv /home/ubuntu/.venvs/dda-portal-tests,
WITH conftest, explicit file list, -q -p no:cacheprovider):
14 files (the 2 new + test_jp6_engine_config_multimodal.py,
test_jp6_fit_check_units.py, test_property_jp6_fit_check_soundness.py,
test_property_jp6_fit_preservation.py, test_jp6_publish_gate_per_arch.py,
test_property_fit_check_decision.py, test_vllm_publish_fit_gate.py,
test_vllm_engine_config_detail_and_audit.py,
test_property_engine_config_update_roundtrip.py,
test_property_engine_config_invalid_updates.py,
test_property_fit_unverified_never_blocks.py,
test_vllm_fit_check_estimation.py)
-> 195 passed, 3 skipped, 0 failed
The 3 skips are the declared [HARDWARE] legs (H6/thor1, H2/three ONNX
co-tenants, 3.11 device state), reported with -rs — never silent.
Security guard trio, host-side: 6 passed, 3 skipped — GREEN (the same
deliberate skips: 2 x "no cdk.out copies present", 1 x the gitignored vendored
edgemlsdk deploy.py).
Documentation only. tasks.md is the only file changed (+256/-0); no source file, no test, no baseline. The portal leg already shipped in b0e8683. Records task 11 OUTCOME blocks 10-18 from the on-device verification session on ryanorinagxdevkithomelabjp622 (LocalServer.arm64JP6 1.0.62). H-tier status at this commit: - H1 SATISFIED. The published component stages gpu_memory_utilization 0.55 with limit_mm_per_prompt {"image": 1, "video": 0} and the model reaches READY; generate answered HTTP 200 earlier the same day. Measured KV 3.75 GiB at 17.14x with co-tenants resident, 6.48 GiB at 29.62x on a near-empty device, against the 1.0.59 baseline of 0.65 GiB at 2.95x. - H2 footprint half SATISFIED and measured: all six Triton stubs resident at 5.84 GiB (cookies-binary 3.82, rf-detr 0.99, yolo 0.89, three marshal 0.14), matching the ~5.7 GiB baseline. None evicted, none on CPU. Inference half OWED - no inference driven through any ONNX model, which is the sole reason task 11 stays open. - H3 substantially verified, H4 verified, H5 not observable at the current healthy KV margin. New defects found and recorded, none fixed here: - Staging collision root cause. Two deployed workflows HARD-depend on the old vLLM component name, so Greengrass is required to keep the stale component installed and it re-stages 0.4 on every lifecycle event. A local --remove reported SUCCEEDED and removed nothing - a silent no-op. - The collision is a concurrent-write race, not ordered last-write-wins: both components staged the same directory 2 ms apart (08:53:59.239 -> .241Z). An owner marker alone is insufficient; the stage needs mutual exclusion, and the guard must cover --cleanup's unload step as well as its rmtree. - Load-order dependence. Proven from one reading: refused with "failed condition(s): starvation" while the budget arm passed by 4.20 GiB (16.48 vs 12.28 required). The starvation arm keys on MemAvailable, a function of which subsystem loaded first, so the preflight verdict is non-deterministic across restarts. Raising gpu_memory_utilization cannot fix a starvation refusal. Both-resident headroom is ~2.00 GiB of 29.96 GiB. - Publish-path naming divergence: record 3df57cf9 carries legacy component_name "model-vllm-qwen2-5-vl-7b-instruct-awq" while published_components[0] is the -jetson-xavier-jp6 suffixed name. Every re-publish orphans workflow HARD dependencies and recreates the collision. Recommended as its own spec - fleet blast radius, not a JP6 memory issue. Corrections to earlier readings in this same file, kept explicit rather than quietly superseding: - LogManager is not dead; it backfills in multi-hour bursts (freshest 04:06:09Z at 09:03Z, caught up to 08:51:16Z by 10:41Z). The four candidate failure mechanisms recorded earlier are retired. - greengrassv2 list-installed-components cannot be trusted to enumerate what is running; it omitted a component that greengrass-cli reported RUNNING. - The 22.12 GiB vLLM footprint and the ONNX READY state in blocks 12 and 15 came from different backend lives and were never co-measured. Remediation applied this session (cloud-side, operator-approved): the component rename was undone. Bare model-vllm-qwen2-5-vl-7b-instruct-awq 3.0.0 was created as a four-field transform of 2.0.0 (version, artifact URI, digest, one Startup substring) carrying the byte-verified good artifact (sha256 2127901... / ISeQFi3c297udsLiGBBmWjVd8wn7+HzIqVD0mDzdAnU=), and deployed as dd209f02-0916-4898-bff5-9a2c403a98a3 with a proven-minimal diff: one component out, one in, 12 carried byte-identical, zero modified. This collapses to a single writer, satisfies the workflows' HARD dependency, and restores the CloudWatch log group without a LogManager map edit. Device state at this commit: vLLM READY, both shadow-reported ONNX models READY on GPU, gpuDegraded false, core device HEALTHY, exactly one vLLM component installed. Rollback documents retained in the untracked .jp6_rename_undo/. Still UNVERIFIED on hardware: H2's inference half, H5, H6 (task 12), H7/H8 (task 14). The five device-leg defects (preflight omits non_torch_memory, the starvation latch samples before async reclaim, unload sets the latch, the thin-margin warning did not fire at 0.77 GiB KV) plus H9's floor semantics and H11/H12's owner marker and mutual exclusion all still need ONE JP6 component build. Nothing in this commit is "done" for an on-device change per .kiro/steering/builds.md.
…yet)
Device-leg code for the SINGLE upcoming aws.edgeml.dda.LocalServer.arm64JP6
build. NOT yet built, NOT yet on hardware: nothing here is "done" for an
on-device change per .kiro/steering/builds.md.
src/backend/vllm_runtime/manager.py and src/backend/dda_triton/vllm_model_prep.py
plus their host tests. _reclaim_gpu_memory and its torch.cuda.is_initialized()
gate are byte-identical; nothing new initialises CUDA in the parent process.
FIVE items, each with production evidence in task 11's OUTCOME blocks:
1. Offline-cache gate (NEW, task 11 block 18). Three consecutive load failures
on transient DNS ("Failed to resolve 'huggingface.co' ([Errno -3] Temporary
failure in name resolution)") at 12:00:47Z / 12:02:09Z / 12:03:22Z drove
currentState=BROKEN, which stranded the two workflows that HARD-depend on
the component at INSTALLED and took the core device UNHEALTHY - with the
repository already staged and the weights already cached. The network call
is vLLM/transformers resolving the repo id, not our code. Fix: when
estimate_weights_on_disk locates the weights, construct the engine with
HF_HUB_OFFLINE=1 / TRANSFORMERS_OFFLINE=1 for the duration of construction
only, restored exactly in a finally (previously-absent keys deleted, never
left as ""). A cache miss is retried exactly once with the originals back,
so a first-time or partial download is never blocked.
2. Retry budget (NEW, same evidence). LOAD_HTTP_ERROR now exits 0, like
LOAD_PREFLIGHT_REFUSED: a MODEL that cannot load must not be able to mark
the COMPONENT broken and take unrelated workflows with it. The prominent
ERROR keeps every element 3.8 pins. LOAD_UNREACHABLE keeps exit 1 - that is
the one case where a component-level retry IS the recovery.
3. H11/H12 owner marker + mutual exclusion. Evidence: two components staged
the same path 2 ms apart (08:53:59.239Z -> .241Z), and a non-owning
Shutdown unloaded and deleted a freshly-loaded model 0.6 s after its load
succeeded (11:00:23.768Z -> 11:00:24.386Z). So ordering alone cannot make
the stage safe and the guard must cover --cleanup's unload as well as its
rmtree. Fix: .dda_stage_owner.json written atomically inside the staged
tree; fcntl.flock on a lock file OUTSIDE it so it survives --cleanup;
foreign teardown refused (both unload and removal skipped) with exit 0;
foreign stage permitted but warned. Every guard fails OPEN - a stuck lock
or corrupt marker must never brick a device.
4. Starvation latch settles (defect (b), block 9 amendment 1). The latch
sampled /proc/meminfo before the driver's asynchronous release, reported
"9.45 GiB did not come back", and ~5.2 GiB came back unaided with no
restart. Fix: a first reading within tolerance returns immediately (healthy
path never sleeps); a starved-looking reading re-samples; it latches only if
every sample is short, and reports the BEST reading with the sample count
and window. STARVATION_SETTLE_DELAY_SECONDS / _RESAMPLES are labelled
PROPOSED, calibration owned by task 14 / H8. Sleep is injectable.
5. Thin-margin visibility (defect (d), block 9). A load reached READY with
0.77 GiB of KV against the 1 GiB floor and nothing warned: kv_bytes was
None and the fallback was an invisible debug line. Fix: three readability
cases. Partial (geometry readable, kv_bytes unknown - the device shape) now
reaches a verdict from concurrency and says the KV size is unknown rather
than guessing. A cache_config that yields nothing usable warns it could not
be VERIFIED. An engine exposing no cache_config at all stays quiet on the
byte-identical debug line, so every pinned suite's fake is unaffected.
Deliberate deviation, endorsed: the offline retry is scoped by
_NO_OFFLINE_RETRY_TOKENS to exclude kv-cache-exhaustion, allocator-nvml-fault
and preflight-refused. Retrying a device-memory failure would spend another
~4 min of doomed profiling starting with less memory than the first attempt -
the cascade defect 1.5 / Decision 5 exist to stop. Two existing tests
(exploration Case 6, Property 6-D) pin one construction per failed KV-OOM load
and caught the unscoped version.
NOT implemented, deliberately: defect (c) as recorded does not match the code.
unload() clears the latch and never sets it; only _fail() latches, and only
with available_before, which the refusal path does not pass. The on-device
observation was real, so the mechanism needs re-deriving rather than a patch
against a mechanism that cannot be reproduced.
Conscious repoints, verbatim originals recorded at each site, nothing weakened.
Six tests pinned the one deliberately changed mapping LOAD_HTTP_ERROR -> 1:
deploy_reliability/test_vllm_prep_error_paths_preservation.py (2),
jp6_vllm_kv_cache_oom test_preservation_jp6_kv_cache_oom.py,
test_property_device_preflight.py, test_property_jp6_device_preservation.py,
test_integration_preflight_prep.py. Every one keeps its LOAD_UNREACHABLE -> 1
arm, and the integration case GAINED assertions that the failure is still
reported in full. Property 7-I repointed from "one debug line" to "one
WARNING" for engines that do expose a cache_config.
Gates, all re-run and verified:
- jp6_vllm_kv_cache_oom 188 passed / 3 skipped (was 159/3); the 3 skips are
still exactly the declared [HARDWARE] H2/H6/3.11 legs
- vllm_runtime + vllm_runtime_tests 29, text_generation 15,
deploy_reliability 72, prep pair with conftest 20 - all at baseline
- vllm_model_reload 44 passed / 1 failed and vllm_hf_cache 15 passed /
1 failed: the SAME two cross-spec vllm_model_prep.py sha256 goldens, owned
by vllm-model-reload-after-backend-restart and vllm-hf-cache-persistence.
New hash c2a02e7577e900bfc7395ac1f7cfbfd8c04c2d62bae359cf4d7f911f125595e1
(goldens still record 9f3b148a...). Deliberately not rebaselined.
- security guard trio 6 passed / 3 skipped GREEN. It came back RED first on
the cdk.out drift guard (unrecorded copies under
asset.921650fa.../{model_converter,components,jwt_authorizer,packaging}.py
regenerated by the portal deploy); cdk.out was moved aside per builds.md
and the trio then matched its recorded baseline. Recorded because that guard
fires AFTER the ~1h compile and would have burned the build.
- No preservation-tracked file touched, so no security rebaseline is needed.
STILL OWED before the build: the required_bytes redesign - defect (a)
non_torch_memory allowance, H9 KV-floor semantics, H8 activation
recalibration and the units model - which moves memory_budget.py,
vllm_fit_check.py and the Property 8 parity test together. Also owed: the
bugfix.md 3.8 amendment for the LOAD_HTTP_ERROR exit-code mapping, and the
load-order/starvation-arm finding from block 18.
…9 + units)
Completes the device leg for the SINGLE upcoming LocalServer.arm64JP6 build.
Both legs move together because Property 8 pins them: memory_budget.py and
vllm_fit_check.py, plus the parity test and the suites that priced the old
arithmetic. NOT yet built, NOT yet on hardware.
THREE constant/semantic changes, all with measured provenance:
(a) NON_TORCH_MEMORY_BYTES = 2 GiB, ADDED to `required`. vLLM subtracts a
non-torch residency from the SAME budget on every load and the shipped
preflight omitted it entirely, modelling three of the four terms - which
made its own "you need at least util = X" advice wrong in the PERMISSIVE
direction and arguably violated 2.1 ("SHALL model every term that consumes
the gpu_memory_utilization budget"). Evidence: at util 0.45 the device
printed "model weights take 6.59GiB; non_torch_memory takes 2.18GiB;
PyTorch activation peak memory takes 4.93GiB; the rest of the memory
reserved for KV Cache is -0.22GiB" - a load that PASSED the shipped
preflight and then failed for real.
2 GiB is the round figure nearest the MEDIAN (2.18 GiB) of seven measured
readings: -0.05, 0.94, 0.98, 2.18, 3.67, 4.76, 8.29 GiB. Deliberately the
middle of the spread, not the 8.29 GiB worst case, because encoding the
worst case as a certainty would refuse the configuration that serves
today. Labelled a PROPOSED THRESHOLD / ESTIMATE; calibration owned by
task 14 / H8. Known limitation stated in the code: a high-non_torch run is
under-predicted, which SHRINKS KV rather than aborting (vLLM computes
KV = budget - weights - non_torch - activation), and the compensating
control is the thin-margin WARNING.
H8 ACTIVATION_WEIGHT_FRACTION 0.75 -> 0.375. The 0.75 came from one point
(4.92 GiB against 6.47 GiB of weights) now understood to have been a
TWO-unit measurement with video unbounded. The measured per-unit pair on
the device, same model and same util 0.55: one unit {"image":1,"video":0}
-> 2.47 GiB; two units {"image":1} -> 4.93 GiB. 2.47/6.59 = 0.375. At
6.45 GiB and one unit that predicts max(2.00, 0.375 x 6.45) = 2.42 GiB
against a measured 2.47 GiB. MULTIMODAL_IMAGE_INCREMENT stays 1.0 - the
2:1 unit step is now MEASURED-confirmed within 0.01 GiB - but the
per-additional-IMAGE direction remains unmeasured and is still an
ESTIMATE.
H9 The KV floor is NO LONGER a term in `required`. MINIMUM_KV_CACHE_BYTES
keeps its 1 GiB value and becomes the thin-margin WARNING threshold applied
to the PREDICTED REMAINDER (new kv_headroom_bytes), which is exactly how
vLLM computes KV: what the budget leaves over. Design Decision 2 always
called it "a serving-margin floor, not a hard load threshold" and Decision
6 always made a sub-floor remainder the Thin_Margin WARNING; the SHIPPED
preflight charged it hard, and that is what refused at util 0.4 the exact
configuration LocalServer 1.0.59 demonstrably SERVED (0.65 GiB of KV at
2.95x concurrency for 4096 tokens). A configuration that fits but leaves
under 1 GiB now PASSES with warnings == ["thin_margin"].
An intermediate version kept a hard KV_VIABILITY_FLOOR_BYTES = 0.25 GiB
between the 0.65 GiB that served and the -0.22 GiB that failed; the simpler
decision (no hard KV term) was taken, and the -0.22 GiB failure is still
not admitted by accident - that load ran video-unbounded, i.e. two units,
which the units model prices at 4.84 GiB of activation, leaving a 0.19 GiB
remainder that trips the thin-margin warning.
units The device mirror now sizes from TOTAL multimodal units (images +
videos) like the portal, so an unauthored limit_mm_per_prompt.video costs a
full extra unit - vLLM's own per-modality default. PROPERTY 8 PARITY IS
THEREFORE EXACT AGAIN: the one-directional portal >= device stopgap
recorded in task 11's ninth OUTCOME block is CLOSED, and the suite now
asserts equality on every mirrored constant, required_bytes,
kv_headroom_bytes, units, co-tenancy, the cap and condition A itself, in
both authoring shapes.
Also: bugfix.md 3.8's exit-code clause amended for the LOAD_HTTP_ERROR ->
exit 0 change, with the superseded text preserved verbatim as a blockquote.
A CORRECTION TO THE PREVIOUS COMMIT'S RECORD. cd855db reported the jp6 dir at
188 passed / 3 skipped. It was actually 184 passed / 4 failed: I verified 188/3,
and a concurrent dispatch then edited test_property_device_preflight.py into a
state pinning intermediate constants (NON_TORCH_ALLOWANCE_BYTES,
KV_VIABILITY_FLOOR_BYTES) that no version of memory_budget.py ever defined,
before I staged it - so its oracle raised AttributeError. I committed a file in
a state I had not tested. Verify-then-commit needs to be one step in a tree
with concurrent dispatches. The 4 are green here.
Conscious repoints: 18 sites across 8 files, each with the superseded assertion
recorded verbatim and the reason named. Several got stronger: the threshold pin
now pins five constants AND the whole composition of `required` over a 3x3 grid
AND the absence of both intermediate names; the viability-floor message check
became the predicted-remainder-vs-serving-margin sentence plus a positive
assertion that no viability floor is named; the thin-margin bracket moved to
the bound the code actually enforces. Nothing deleted, nothing skipped,
nothing loosened. S9 untouched with zero diff.
Gates, all re-run and verified by me:
- jp6_vllm_kv_cache_oom 190 passed / 3 skipped / 0 failed in 30s; the 3 skips
are still exactly the declared [HARDWARE] H2/H6/3.11 legs
- Property 8 parity inside that run, exact
- portal, the 13 required files WITH conftest: 158 passed / 3 skipped - the
recorded baseline exactly. S9 zero diff.
- vllm_runtime + vllm_runtime_tests + text_generation + deploy_reliability:
116 passed = 29 + 15 + 72
- vllm_model_reload + vllm_hf_cache: 2 failed / 59 passed - the SAME two
cross-spec vllm_model_prep.py sha256 goldens owned by
vllm-model-reload-after-backend-restart and vllm-hf-cache-persistence.
Deliberately not rebaselined.
- security guard trio 6 passed / 3 skipped GREEN
- no preservation-tracked file touched, so no security rebaseline needed
STILL UNVERIFIED ON HARDWARE (H1-H8) and unbuilt. The load-order finding from
task 11 block 18 - that the starvation arm keys on MemAvailable and so the
preflight verdict is non-deterministic across restarts - is NOT addressed here
and remains open alongside H7 and the H8 calibration.
…only, block 19 + task 14 addendum)
…ied on 1.0.63, offline gate engaged, H8 reading 6 (docs only)
…n, R15 IAM scoping
…n, R15 IAM scoping
… code Device verification pending. Committed to remediation at the owner's request so the build fleet can build the LocalServer components for the on-device checks that .kiro/steering/builds.md requires. Keep this commit off integration/all-specs until those checks pass; record the results in .kiro/specs/security-scan-remediation-high/tasks.md.
An account-level SSM State Manager patch association runs AWS-RunPatchBaseline (Install, RebootIfNeeded) on every new instance at registration; the patch run exits 194 and the SSM agent reboots the runner ~66 s later, 1-3 minutes into the build (BuildKit EOF). The ephemeral gate only checked SSM Online plus the bootstrap marker, which is written before the patch run finishes. P0-A: after the marker is READY, provision_ephemeral reads DescribeInstanceAssociationsStatus (runner_association_statuses) and build_planner.decide_host_settled waits while any association is Pending/InProgress or the read fails, inside the same bootstrap budget; strictly past it the job fails BOOTSTRAP_TIMEOUT naming the associations and the runner is terminated. bootstrap.settled_at and bootstrap.associations are recorded. IAM: a read-only ssm:DescribeInstanceAssociationsStatus (*) statement in grantSsmCommands. P0-B: the runner user data opens with a marker guard, its own flock and a root 0700 self-install as a cloud-init per-boot script, so a reboot mid-bootstrap re-runs it; the marker write stays last. Every apt-get in the runner and fleet user data and setup-build-server.sh carries -o DPkg::Lock::Timeout=600. P1: verify_and_start_dedicated defers (never fails) a dispatch while an association runs on the server, capped at 30 minutes from the deferral start, then dispatches with an advisory; an unreadable status does not defer (fail open). The deferral writes deferred_at, so the 5-minute cadence and the lock-deferral stall backstop are unchanged. Tests (docker dda-portal-test:py311 as uid 1000, HYPOTHESIS_PROFILE=ci): - portal_builds: baseline 6 failed / 1222 passed / 1 skipped, after 6 failed / 1263 passed / 1 skipped. Same 6 pre-existing failures (test_build_diagnostic_api x4, test_ref_aware_bootstrap_property x2). - new test_runner_settle_unit.py: 31 passed; new P0-A/P1 tick scenarios, gate properties and executed per-boot re-entry tests pass. - jest build-fleet-stack: 22 -> 23 passed; full npm test: 305 -> 306 passed, 28 suites. - guard pair 4 passed / 3 skipped and IAM CDK synth gate 17 passed, both unchanged; no security baseline pins an edited file. - bash -n clean on setup-build-server.sh and the generated user data; shellcheck findings unchanged (runner user data: none).
bugfix.md with the owner decisions (P0-A, P0-B, P1 approved by Ryan Vanderwerf on 2026-10-07; P2 deferred), design.md, tasks.md and verification-notes.md for the fix in 58524e5. The notes record the suites run before and after the change (portal builds 6 failed / 1222 passed -> 6 failed / 1263 passed, the same 6 pre-existing failures; jest 305 -> 306 passed; guard pair and IAM CDK synth gate unchanged), the shell checks, the cloud-init module order read from the Ubuntu 22.04 and 24.04 packages, and the apt lock finding: DPkg::Lock::Timeout covers the dpkg lock but not the lists lock that apt-get update takes. Deploy evidence is left for the deploy step.
The use-case bucket configuration needs the S3 PutBucketAbac/GetBucketAbac operations and the S3 Control TagResource/UntagResource/ListTagsForResource bucket-tagging operations, which the Lambda-managed runtime's bundled SDK may predate. boto3/botocore 1.43.101 with s3transfer, jmespath, python-dateutil and six are installed with pip --no-deps so the vendored urllib3 2.8.0 and requests trees are untouched; shared_utils logs the layer's botocore version once per cold start and test_shared_layer_sdk_operations.py pins the version and the operation names.
…red layer shared_utils gains the membership and creation grammars, parse_s3_uri, the BucketConfigError hierarchy with the ten REMEDY_* texts (S3 Control tag-resource form) and bucket_config_error_response, portal_owned_buckets, allowed_s3_buckets, assert_bucket_allowed (now the first statement of get_s3_client_for_bucket; the role choice moves verbatim into _select_s3_role_for_bucket, signature and the fourteen call sites unchanged), EligibilityResult with shared_with_usecases / stale_claim, probe_bucket_eligibility (retry only on request, one PermanentRedirect region retry) and check_bucket_attachable (portal-owned first, grammar, configuration Scan, probe or evidence, tag claim with the stale-claim remedy, PortalAdmin override). compute-stack.ts hoists the UseCases role and grants it s3:CreateBucket, s3:TagResource conditioned on aws:RequestTag/dda-portal:created-by=portal and s3:PutBucketABAC on arn:aws:s3:::*, pinned by usecase-bucket-config-iam.test.ts; the three statements are recorded in iam_post_fix_approved_additions.json (the fixed baseline template stays untouched, as test_preservation_iam_cdk_synth.py requires). test_usecase_bucket_config_units.py covers the primitives and carries the writer guard, asserted against the pre-change writer set until the use case API feature adds the list helpers; fixtures/bucket_attribute_writers.py checks the guard rule shape by shape.
…ckets
Use case record and API for the configured-bucket set (design §5):
- usecases.py: `_add_additional_bucket` / `_remove_additional_bucket`
(attribute_exists on every list write, ReturnValuesOnConditionCheckFailure
tells "item gone" from "already present"); GET /usecases/{id} returns
`allowed_s3_buckets` and `additional_s3_buckets`; PUT /usecases/{id}
validates every bucket entering the record through
`check_bucket_attachable` (changed configured buckets, add/remove deltas
capped at 10 names per request and 100 per use case, Clear semantics for
data_s3_bucket, `rejected` per failing name, nothing written on refusal,
404 and no phantom record for an unknown or deleted id); POST /usecases
validates s3_bucket / data_s3_bucket with the tag read-after-write retry
and an optional additional list without it, and `tag_bucket_for_portal`
is read-only; POST /s3-buckets creates a portal-managed bucket in the
order create_bucket(CreateBucketConfiguration.Tags) -> put_bucket_abac ->
read-after-write get_bucket_tagging -> one s3control tag_resource
fallback -> auto-add with the read tags as evidence (409 on an owned
name, 500 bucket_tagging_failed, add_error paths); GET /s3-buckets marks
`eligible` / `portal_created` from one tagging query.
- data_management.configure_data_account: record load (404), validator for
a changed non-Clear data bucket, attribute_exists on the write.
- shared_utils: `read_bucket_tags` (the one bounded tag read behind the
probe and the bucket creation) and `audit_bucket_allowed` (the uniform
s3_bucket_allowed row every writer emits after its write, carrying the
validator's shared_with_usecases / stale_claim).
- Tests: test_usecase_bucket_config_api.py (moto, 54 cases incl. the call
order, the tag_resource fallback, exact Tags arguments, bulk add without
sleeps, bounded creation retries, one audit-row test per writer),
test_property_bucket_attach_eligibility.py (Property 9), the writer guard
switched to the five writers and two helper call sites; the sink-step
test files rebind the live shared layer module in their fixtures so they
are order-independent; existing tests that created use cases on absent
buckets or stored bare few-shot references updated to the feature's
contract.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
… the integration branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings
mainup to date withintegration/all-specs: 355 commits and 142 specs since #68. Each spec lives under.kiro/specs/<name>/with its requirements, design, tasks and verification record.Highlights:
Upgrade notes
See
edge-cv-portal/CHANGELOG.md. In particular:dda-portal:managed=truetag and the Allow for this use case action. Other buckets return 400bucket_not_allowed.POST /models/validateandPOST /models/inspectrequire the DataScientist role, likePOST /models/import.PORTAL_REGISTRY_ENFORCED=true ./deploy-infrastructure.shafter the registry backfill. The deploy script now warns when enforcement is off.Testing
verification-notes.mdortasks.md: unit and property tests, plus on-device checks on JetPack 5, 6 and 7 devices and an x86 station for edge changes (.kiro/steering/builds.md).Not included
spec/jpeg-folder-source-decode) is waiting on device verification and will follow separately.