Skip to content

Added the demo set generator and the EMBER upload workflow - #27

Merged
CodyCBakerPhD merged 17 commits into
mainfrom
claude/inspect-tab-demo-ffmpeg-58vjr7
Aug 23, 2026
Merged

Added the demo set generator and the EMBER upload workflow#27
CodyCBakerPhD merged 17 commits into
mainfrom
claude/inspect-tab-demo-ffmpeg-58vjr7

Conversation

@CodyCBakerPhD

@CodyCBakerPhD CodyCBakerPhD commented Aug 22, 2026

Copy link
Copy Markdown
Member

scripts/generate-demos.sh derives 28 short videos from a source recording (supplementary Video S1 of PLOS ONE 10.1371/journal.pone.0025390), each varying exactly one thing the app surfaces: atom layout (faststart, fragmented MP4), container (MOV, MKV, WebM, AVI), codec and profile (H.264 Baseline / High 10 / High 4:4:4, HEVC, AV1, VP8/VP9, MJPEG, lossless FFV1), GOP length, B-frame structure, bitrate behaviour (true CBR, starved cap), track properties (display-matrix rotation, VFR, half resolution, no audio) and metadata tags (rich vs stripped). The output is a BEP047 dataset (bids-specification#2231, as clip-extractor#41 writes it): one session per variant at sub-01/ses-<label>/beh/sub-01_ses-<label>_video.<ext>, a JSON sidecar per video carrying ffprobe-extracted technical keys (including ImageBitDepth and VideoCodecRFC6381 where determinable), and a dataset_description.json at the root recording the CC-BY-4.0 license, the PLOS article as SourceDatasets, the generation date, and the per-session index under an encoding-helper key. Each file also explains itself in its title/comment tags for the planned replacement of the Load Sample button.

The unmodified recording is ses-original, the first of the 29 sessions: the dataset is raw at its outer level, so it needs no sourcedata/ layer of its own. It is not in this repository — a video blob would sit in the git history forever — and the generator fetches it from that archived session when there is no local copy.

scripts/upload-demos.sh copies the dataset into EMBER dandiset 000527 with the dandi CLI (validation skipped, since these are plain videos); dandiset, instance and layout are fixed. The manual Upload demos to EMBER workflow dispatches with no parameters and runs both, keeping the set as a run artifact. It needs an EMBER_DANDI_API_KEY repository secret with write access to the dandiset.

No version bump: nothing under src/, configs/ or the package files changed.

Prompts

I'd like to setup a range of demos showcasing the importance of each aspect of the inspect tab; can you use the attached source video in a simple bash script using ffmpeg to create a range of file structures, both H264 and otherwise (also useful for interoperability testing) and a separate script that pushes them via a workflow dispatch github action to ember dandiset https://dandi.emberarchive.org/dandiset/000527 ?

A follow up would then expose direct integration that would replace teh load sample button

@"/root/.claude/uploads/205159ee-a1e4-5de0-a73a-2e48a585909c/a6f51fbb-Video_S1.m4v"

rephrase away from "Inspect Demos" to "Upload demos" (and just 'demos' elsewhere, they are not unique to the inspect tab

the dandiset ID will not change, no need to expose that as a parameter

the dandi instance will not change, no need to expose that as a parameter

the point of the workflow is to upload so remove that checkbox option too

might as well clamp the directory structure; follow BEP047 like you do for output of clip-extractor, refer to this as 'sub-01' and add different desc- to each

for BEP047 (again refer to brain-bbqs/clip-extractor#41 for specifics) compatability in this output, replace manifest.json with a dataset_description.json and extract metadata sidecars for each video. Actually, maybe replaces 'desc-' with 'ses-' as if each video case was a different session rather than a variant recording/acquisition of a single session and therefore nest each under its own level

don't forget "ImageBitDepth" and "VideoCodecRFC6381" fields

also drop the section [encoding-helper sidecar block] from each video sidecar, that info belongs more in the dataset_description.json

should probably include the date the dataset was generated on in the "Description" of "GeneratedBy"

also make license CC-BY (or copy w/e license is from source https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0025390#s5) also list that as source dataset

can you add the name for the source dataset? any other useful metadata to include for that?

"from the ses-reference recording" -> "from the original video"

IDT the version should be the year; also order keys by "Name" -> "Description" then as-is

ok last thing; now that it's all on the archive, remove the video from this branch so it doesn't get injected into the git blob history

yeah i mean if its not already on there somewhere in the sessions go ahead and add it and adjust the upload script to operate on that remote blob (ill make it persistent)

unpack the sourcedata/rawbids, the entire dataset at the outer level is raw. just include that one as the 'ses-original'

🤖 Generated with Claude Code

https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn

scripts/generate-inspect-demos.sh derives 28 short videos from the
committed source recording (scripts/data/Video_S1.m4v), each varying one
aspect the Inspect tab surfaces: atom layout (faststart, fragmentation),
container (MOV/MKV/WebM/AVI), codec and profile (H.264 Baseline/High
10/High 4:4:4, HEVC, AV1, VP8/9, MJPEG, FFV1), GOP and B-frame
structure, bitrate behaviour (CBR, starved cap), track properties
(rotation, VFR, resolution, no audio) and metadata tags. Each file
carries its own explanation in its title/comment tags, and a
manifest.json describes the set for future in-app integration.

scripts/upload-inspect-demos.sh pushes a generated set to a dandiset on
the EMBER archive with the dandi CLI, and the Inspect demos workflow
(workflow_dispatch) runs both against EMBER dandiset 000527, keeping the
set as a run artifact and supporting a dry run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
@github-actions

github-actions Bot commented Aug 22, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://brain-bbqs.github.io/encoding-helper/pr-preview/pr-27/

Built to branch gh-pages at 2026-08-23 20:09 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

Comment thread .github/workflows/upload-demos.yml Fixed
The job only checks out the repo; the EMBER upload uses its own key.
Flagged by CodeQL on #27.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@CodyCBakerPhD CodyCBakerPhD self-assigned this Aug 23, 2026
@CodyCBakerPhD
CodyCBakerPhD marked this pull request as ready for review August 23, 2026 00:51
The demos exercise the whole app, not just one tab: the workflow is now
"Upload demos to EMBER" (upload-demos.yml), the scripts are
generate-demos.sh and upload-demos.sh, and the default asset prefix is
demos/.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
@CodyCBakerPhD CodyCBakerPhD changed the title Added the Inspect-tab demo set generator and the EMBER upload workflow Added the demo set generator and the EMBER upload workflow Aug 23, 2026
CodyCBakerPhD and others added 14 commits August 22, 2026 21:03
Signed-off-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com>
The dandiset (000527), instance (ember-dandi) and layout are no longer
workflow inputs: the workflow now dispatches with no parameters and the
demo files land as sub-01/video/sub-01_desc-<label>_video.<ext>, BIDS
entity naming with a distinct desc- per variant, the way clip-extractor
names its output. The manifest sits beside the videos.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The dandi download of the dandiset record silently produced nothing on
the runner, and with the target directory then missing, cp collapsed
the sub-01/ level into it. dandi upload only reads the identifier out
of dandiset.yaml, so the script now writes that one line itself and
copies the demo tree into a directory that is guaranteed to exist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The dandi CLI reads the key for a given instance from
{INSTANCE_NAME}_API_KEY, so for ember-dandi it ignored DANDI_API_KEY
and fell back to an interactive prompt, which EOFs in CI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
dandi upload only considers extensions it recognizes as assets, which
silently left the two .webm files and manifest.json behind; DANDI_DEVEL
with --allow-any-path uploads the whole tree.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
Each variant now lands as its own session under the beh datatype
directory (sub-01/ses-<label>/beh/sub-01_ses-<label>_video.<ext>),
following bids-standard/bids-specification#2231 as clip-extractor writes
it. A JSON sidecar next to each video carries BEP047's technical keys as
ffprobe extracts them plus the demo detail under an encoding-helper key,
and a dataset_description.json at the root replaces manifest.json.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
Sidecars gain ImageBitDepth and, where one can be determined (via
ffmpeg's DASH muxer, or built from the level for HEVC),
VideoCodecRFC6381; the per-session demo detail moves out of them into
dataset_description.json under the encoding-helper key, so one fetch
indexes the set. The dataset also now records its CC-BY-4.0 license,
the PLOS ONE article the source video is supplementary material of as
SourceDatasets, and the generation date in GeneratedBy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The entry now carries the article title, authors, year, license and a
note that the source recording is its Supporting Information Video S1;
the dataset Description says "the original video" instead of naming the
reference session.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The publication year moved into the Description; the year is not a
dataset version.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The recording the demos are derived from now travels with them, in the
place BIDS keeps a raw original (with its own dataset_description.json,
as clip-extractor's rawbids copy has), and the generator fetches it from
there when there is no local copy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The archive now holds it under sourcedata/rawbids/, so the generator
fetches it from there and no video blob has to live in the git history.
scripts/data/ is ignored, and the workflow passes the archive key to the
generate step so the fetch works against a non-public dandiset.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The dataset is raw at its outer level, so the unmodified recording needs
no sourcedata/rawbids/ layer of its own: it is simply the first session,
with the same sidecar every other session gets and an index entry that
carries no ffmpeg_args, since no ffmpeg call produced it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
The archive still holds the recording where it sat before it became
ses-original, and a runner has no local copy to publish the new path
from. The fallback lets the first run after the move find it, and comes
out once that run has published ses-original.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn
@CodyCBakerPhD
CodyCBakerPhD merged commit 7276f27 into main Aug 23, 2026
16 checks passed
@CodyCBakerPhD
CodyCBakerPhD deleted the claude/inspect-tab-demo-ffmpeg-58vjr7 branch August 23, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants