Added the demo set generator and the EMBER upload workflow - #27
Merged
Conversation
scripts/generate-inspect-demos.sh derives 28 short videos from the committed source recording (scripts/data/Video_S1.m4v), each varying one aspect the Inspect tab surfaces: atom layout (faststart, fragmentation), container (MOV/MKV/WebM/AVI), codec and profile (H.264 Baseline/High 10/High 4:4:4, HEVC, AV1, VP8/9, MJPEG, FFV1), GOP and B-frame structure, bitrate behaviour (CBR, starved cap), track properties (rotation, VFR, resolution, no audio) and metadata tags. Each file carries its own explanation in its title/comment tags, and a manifest.json describes the set for future in-app integration. scripts/upload-inspect-demos.sh pushes a generated set to a dandiset on the EMBER archive with the dandi CLI, and the Inspect demos workflow (workflow_dispatch) runs both against EMBER dandiset 000527, keeping the set as a run artifact and supporting a dry run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
|
The job only checks out the repo; the EMBER upload uses its own key. Flagged by CodeQL on #27. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
Codecov Report✅ All modified and coverable lines are covered by tests. 🚀 New features to boost your workflow:
|
CodyCBakerPhD
marked this pull request as ready for review
August 23, 2026 00:51
The demos exercise the whole app, not just one tab: the workflow is now "Upload demos to EMBER" (upload-demos.yml), the scripts are generate-demos.sh and upload-demos.sh, and the default asset prefix is demos/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
Signed-off-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com>
The dandiset (000527), instance (ember-dandi) and layout are no longer workflow inputs: the workflow now dispatches with no parameters and the demo files land as sub-01/video/sub-01_desc-<label>_video.<ext>, BIDS entity naming with a distinct desc- per variant, the way clip-extractor names its output. The manifest sits beside the videos. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The dandi download of the dandiset record silently produced nothing on the runner, and with the target directory then missing, cp collapsed the sub-01/ level into it. dandi upload only reads the identifier out of dandiset.yaml, so the script now writes that one line itself and copies the demo tree into a directory that is guaranteed to exist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The dandi CLI reads the key for a given instance from
{INSTANCE_NAME}_API_KEY, so for ember-dandi it ignored DANDI_API_KEY
and fell back to an interactive prompt, which EOFs in CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
dandi upload only considers extensions it recognizes as assets, which silently left the two .webm files and manifest.json behind; DANDI_DEVEL with --allow-any-path uploads the whole tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
Each variant now lands as its own session under the beh datatype directory (sub-01/ses-<label>/beh/sub-01_ses-<label>_video.<ext>), following bids-standard/bids-specification#2231 as clip-extractor writes it. A JSON sidecar next to each video carries BEP047's technical keys as ffprobe extracts them plus the demo detail under an encoding-helper key, and a dataset_description.json at the root replaces manifest.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
Sidecars gain ImageBitDepth and, where one can be determined (via ffmpeg's DASH muxer, or built from the level for HEVC), VideoCodecRFC6381; the per-session demo detail moves out of them into dataset_description.json under the encoding-helper key, so one fetch indexes the set. The dataset also now records its CC-BY-4.0 license, the PLOS ONE article the source video is supplementary material of as SourceDatasets, and the generation date in GeneratedBy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The entry now carries the article title, authors, year, license and a note that the source recording is its Supporting Information Video S1; the dataset Description says "the original video" instead of naming the reference session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The publication year moved into the Description; the year is not a dataset version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The recording the demos are derived from now travels with them, in the place BIDS keeps a raw original (with its own dataset_description.json, as clip-extractor's rawbids copy has), and the generator fetches it from there when there is no local copy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The archive now holds it under sourcedata/rawbids/, so the generator fetches it from there and no video blob has to live in the git history. scripts/data/ is ignored, and the workflow passes the archive key to the generate step so the fetch works against a non-public dandiset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The dataset is raw at its outer level, so the unmodified recording needs no sourcedata/rawbids/ layer of its own: it is simply the first session, with the same sidecar every other session gets and an index entry that carries no ffmpeg_args, since no ffmpeg call produced it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The archive still holds the recording where it sat before it became ses-original, and a runner has no local copy to publish the new path from. The fallback lets the first run after the move find it, and comes out once that run has published ses-original. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/generate-demos.shderives 28 short videos from a source recording (supplementary Video S1 of PLOS ONE 10.1371/journal.pone.0025390), each varying exactly one thing the app surfaces: atom layout (faststart, fragmented MP4), container (MOV, MKV, WebM, AVI), codec and profile (H.264 Baseline / High 10 / High 4:4:4, HEVC, AV1, VP8/VP9, MJPEG, lossless FFV1), GOP length, B-frame structure, bitrate behaviour (true CBR, starved cap), track properties (display-matrix rotation, VFR, half resolution, no audio) and metadata tags (rich vs stripped). The output is a BEP047 dataset (bids-specification#2231, as clip-extractor#41 writes it): one session per variant atsub-01/ses-<label>/beh/sub-01_ses-<label>_video.<ext>, a JSON sidecar per video carrying ffprobe-extracted technical keys (includingImageBitDepthandVideoCodecRFC6381where determinable), and adataset_description.jsonat the root recording the CC-BY-4.0 license, the PLOS article asSourceDatasets, the generation date, and the per-session index under anencoding-helperkey. Each file also explains itself in itstitle/commenttags for the planned replacement of the Load Sample button.The unmodified recording is
ses-original, the first of the 29 sessions: the dataset is raw at its outer level, so it needs nosourcedata/layer of its own. It is not in this repository — a video blob would sit in the git history forever — and the generator fetches it from that archived session when there is no local copy.scripts/upload-demos.shcopies the dataset into EMBER dandiset 000527 with the dandi CLI (validation skipped, since these are plain videos); dandiset, instance and layout are fixed. The manualUpload demos to EMBERworkflow dispatches with no parameters and runs both, keeping the set as a run artifact. It needs anEMBER_DANDI_API_KEYrepository secret with write access to the dandiset.No version bump: nothing under
src/,configs/or the package files changed.Prompts
🤖 Generated with Claude Code
https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn