Skip to content

feat: opt-in 4:4:4 chroma for SDR MP4 masters - #5820

Merged
jrusso1020 merged 19 commits into
heygen-com:mainfrom
PaprikaChen:feat/chroma-subsampling-444
Nov 11, 2026
Merged

jrusso1020 merged 19 commits into
heygen-com:mainfrom
PaprikaChen:feat/chroma-subsampling-444

Conversation

@PaprikaChen

@PaprikaChen PaprikaChen commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

What

Adds an opt-in chromaSubsampling: "420" | "444" render option (--chroma-subsampling on hyperframes render, a RenderConfig field, and the Studio / producer HTTP render routes). "444" renders an SDR MP4 master with full-resolution colour: libx264 yuv444p (High 4:4:4 Predictive) encoded from lossless PNG capture. The default stays "420" and the default render path is unchanged.

Why

Every SDR MP4 is 4:2:0, which softens or fringes thin coloured text, UI strokes and saturated edges. Motion-graphics masters are mostly that. 4:4:4 keeps the colour, but many players cannot decode it, so it is opt-in only.

Related work

Fixes #5555. The approach was posted on the issue before implementing.

Measurement (issue step 1)

Each clip was encoded from a lossless RGB master with libx264 medium at the same CRF, using the engine's SDR colour conversion (BT.709, aq-mode=3). PSNR and SSIM are computed per plane against the master. Windows 11, ffmpeg 9.0.2, 1920x1080, 30 fps, 5 s.

Clip (CRF 18) Chroma Size (KB) Encode (s) PSNR U / V (dB) SSIM U / V
Coloured text on dark (rendered by HyperFrames, PNG capture) 4:2:0 1756 0.83 38.28 / 37.17 0.9880 / 0.9881
4:4:4 757 1.17 54.44 / 54.01 0.9996 / 0.9997
UI screen recording (Studio window) 4:2:0 1200 0.92 40.76 / 38.20 0.9900 / 0.9874
4:4:4 1064 1.23 53.37 / 53.12 0.9987 / 0.9990
Footage (NASA RED 4K clip, public domain, downscaled) 4:2:0 2496 1.92 51.33 / 51.93 0.9949 / 0.9956
4:4:4 2566 2.45 51.76 / 52.53 0.9950 / 0.9957
Gradient 4:2:0 952 1.43 54.54 / 54.01 0.9971 / 0.9969
4:4:4 984 1.69 53.82 / 53.53 0.9968 / 0.9965

CRF 23 shows the same pattern. Large gain for text and UI (about +13 to +16 dB chroma, visibly cleaner edges, smaller files at the same CRF); none for footage or gradients.

4x nearest-neighbour crops (master | 4:2:0 | 4:2:0 lanczos | 4:4:4):

text-red-on-blue ui-preview-text footage-sphere-handle gradient-edge

End to end through this PR (hyperframes render vs hyperframes render --chroma-subsampling 444, compared with a PNG-capture reference): chroma PSNR 21.9 / 22.9 dB vs 44.6 / 45.4 dB on a 1 px red/blue stripe test.

How

  • Capture is the catch. At JPEG quality 100, Chrome's beginFrame (the Linux default) and CDP screenshots are already 4:2:0; only drawElement's canvas JPEG is 4:4:4, and drawElement can fall back to screenshot mid-render. Verified with Chrome 152 on Windows and on Ubuntu 24.04 (GitHub Actions). So "444" captures PNG on every path and turns drawElement off (compile gate chroma_444).
  • engine: uses CaptureOptions.transparent / capturesAlpha from fix(producer): keep the page background opaque when motion blur renders an MP4 #6262 to capture opaque PNG, so the page background stays in a 4:4:4 MP4. Opaque PNG skips drawElement (JPEG-only verification and worker encode); since fix(producer): keep the page background opaque when motion blur renders an MP4 #6262 that also covers motion-blur MP4, which now captures opaque PNG too. getEncoderPreset takes an optional chromaSubsampling that only changes the SDR MP4 branch to yuv444p; x264 picks High 4:4:4 Predictive and odd sizes need no padding.
  • producer: RenderConfig.chromaSubsampling; resolveCaptureSurface gives the probe and main capture the same format, quality and background; the preset gets the chroma value, so every encode path and the segment-resume hash see yuv444p.
  • Validation: "444" with a non-mp4 format, GPU encoding or HDR output is rejected before any work starts; auto-detected HDR sources fail before capture with a --sdr hint; distributed / Lambda / Cloud Run requests refuse it (chunk workers and the CFR assemble are 4:2:0). The CLI, the Studio route (400) and the producer HTTP server reject the same combinations.
  • GPU encoders: only NVENC lists yuv444p (QSV and VAAPI are NV12-only here, AMF has no 4:4:4 H.264, VideoToolbox H.264 has no 4:4:4 profile), so GPU encoding is rejected rather than silently falling back to 4:2:0 on some hosts. NVENC support could be a follow-up.

Player compatibility (issue step 3)

Also in the rendering guide.

Player 4:4:4 MP4 How tested
Chrome 155 (Windows) Plays Automated: load / play / seek, decoded frame vs ffmpeg 73.9 dB
Edge 155 (Windows) Plays Automated (73.9 dB) and manual
Firefox 157 (Windows) Does not play MediaError 3: no YUV 4:4:4 decoder; canPlayType empty
VLC 3.0.24 (Windows) Plays Automated, frames inspected
Windows Media Player / Media Player app Does not play Manual: "unsupported encoding settings"
Safari, iOS 26.6 Plays Manual
Safari 26.6.2 (macOS 26.6.2) Plays Automated on a GitHub macOS runner (safaridriver): load / play / seek, frame matches visually
QuickTime / AVFoundation (macOS 26.6.2) Plays Automated through AVFoundation (QuickTime Player's playback framework): playable, profile 244, all frames decoded, frame vs ffmpeg 65.5 dB

Every player above plays the default 4:2:0 output.

Cost

Median of 3, 1 capture worker, 1920x1080, 300 frames, Windows 11 with a hardware GPU.

Composition Default Default, drawElement off 444
Coloured text (capture / total) 11.2 s / 16.2 s 20.5 s / 24.9 s 26.8 s / 31.5 s
Full-frame footage + title (capture / total) 19.6 s / 25.9 s 26.0 s / 30.7 s 85.1 s / 91.2 s

About 2x the default render time for text and UI, about 3.5x for full-frame footage: PNG encoding cost grows with image detail. Documented in the guide.

Out of scope (noted, not changed)

  • The default 4:2:0 path downsamples chroma with flags=neighbor; lanczos gains about 1.5 to 2 dB chroma on text and UI with no compatibility cost. Happy to open a separate issue.
  • Opaque PNG capture keeps Chrome's optimizeForSpeed off (it protects alpha); enabling it for opaque PNG may cut the 444 capture cost.

Test plan

  • Unit tests added/updated: engine (getEncoderPreset, encoder args at odd sizes, verification-page background), producer (chromaSubsampling gates, capture surface, compile gate, render request, HTTP server), studio-server (encoder options, render route), CLI (plan, Docker argv, local render forwarding)
  • Manual testing performed: real renders with and without --chroma-subsampling 444 (yuv444p / High 4:4:4 Predictive, background unchanged, PNG capture logged), 4 workers, odd 1281x721 size, every rejection case; player tests above
  • Documentation updated: rendering guide (4:4:4 masters, compatibility, cost), CLI reference, producer docs and README, CLI built-in docs, hyperframes-cli skill reference and manifest
  • Comments follow CONTRIBUTING.md "Comments"; check-comment-citations and comment-ratchet pass

Developed on Windows: engine, producer, studio-server and CLI suites show no new failures against main; the remaining failures there are Windows-only and also fail on main (symlink permissions, POSIX sleep/echo in tests, path separators, ffmpeg 8+ dropping -vsync).

🤖 Generated with Claude Code

@mintlify

mintlify Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit bdd3399.

Approve preview

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the careful work here, and for measuring before building. The tables and the player matrix make the trade-off easy to judge.

I read the engine, producer, CLI and studio-server changes end to end. The design holds up:

  • Capture format and background are decided once in resolveCaptureSurface, and both the probe session and the main capture use it, so the reused probe session can't disagree with the encoder.
  • Every place in the engine that used "PNG" to mean "transparent" now asks wantsTransparentBackground, including the drawElement fallback and the static verification page, so alpha output and motion blur keep their current behavior.
  • The pixel format comes from the preset on every encode path. The encode-while-capturing path, which hardcodes JPEG input, is already gated to JPEG capture, so 4:4:4 can't reach it. The segment-resume hash includes the pixel format, so a resumed 4:4:4 run won't reuse 4:2:0 segments.
  • The rejections line up across the CLI, the Studio route, the producer HTTP server and the orchestrator, and auto-detected HDR fails before capture starts.

The touched unit tests pass at this head on Linux (engine, producer, studio-server and CLI). I did not run an end-to-end render.

Two small things:

  1. The rendering guide says 4:4:4 is rejected for Lambda and Cloud Run renders. Distributed requests built from a render request are rejected, but the Lambda and Cloud Run SDK configs have no chroma field, and validateDistributedRenderConfig ignores keys it doesn't know. A plain JavaScript caller passing chromaSubsampling: "444" would get an ordinary 4:2:0 file with no error. Either reword that line to "not available on Lambda or Cloud Run", or have the shared validator reject the key.
  2. The tests cover each helper, but nothing checks that the orchestrator passes chromaSubsampling through. If a later change drops it from the getEncoderPreset call or the capture surface, 4:4:4 renders would quietly come out 4:2:0 and every test would still pass. One small fixture rendered with "444" and checked with ffprobe for yuv444p, or a test on the encoder arguments the orchestrator builds, would catch that.

Before merge: the full CI suite hasn't run on this branch yet (a maintainer needs to approve the workflow run), and main requires signed commits, so the commits here will need signing. The motion-blur background issue you listed under out of scope looks like a real bug and is probably worth its own issue.

Verdict: COMMENT
Reasoning: No code blockers found and the touched tests pass locally, but the required CI suite has not run at this head and the commits are unsigned, so it is not ready to approve yet.

@PaprikaChen
PaprikaChen force-pushed the feat/chroma-subsampling-444 branch from 90da21a to 7b1241d Compare October 24, 2026 22:03
@PaprikaChen

Copy link
Copy Markdown
Contributor Author

Thanks for the careful work here, and for measuring before building. The tables and the player matrix make the trade-off easy to judge.

I read the engine, producer, CLI and studio-server changes end to end. The design holds up:

  • Capture format and background are decided once in resolveCaptureSurface, and both the probe session and the main capture use it, so the reused probe session can't disagree with the encoder.
  • Every place in the engine that used "PNG" to mean "transparent" now asks wantsTransparentBackground, including the drawElement fallback and the static verification page, so alpha output and motion blur keep their current behavior.
  • The pixel format comes from the preset on every encode path. The encode-while-capturing path, which hardcodes JPEG input, is already gated to JPEG capture, so 4:4:4 can't reach it. The segment-resume hash includes the pixel format, so a resumed 4:4:4 run won't reuse 4:2:0 segments.
  • The rejections line up across the CLI, the Studio route, the producer HTTP server and the orchestrator, and auto-detected HDR fails before capture starts.

The touched unit tests pass at this head on Linux (engine, producer, studio-server and CLI). I did not run an end-to-end render.

Two small things:

  1. The rendering guide says 4:4:4 is rejected for Lambda and Cloud Run renders. Distributed requests built from a render request are rejected, but the Lambda and Cloud Run SDK configs have no chroma field, and validateDistributedRenderConfig ignores keys it doesn't know. A plain JavaScript caller passing chromaSubsampling: "444" would get an ordinary 4:2:0 file with no error. Either reword that line to "not available on Lambda or Cloud Run", or have the shared validator reject the key.
  2. The tests cover each helper, but nothing checks that the orchestrator passes chromaSubsampling through. If a later change drops it from the getEncoderPreset call or the capture surface, 4:4:4 renders would quietly come out 4:2:0 and every test would still pass. One small fixture rendered with "444" and checked with ffprobe for yuv444p, or a test on the encoder arguments the orchestrator builds, would catch that.

Before merge: the full CI suite hasn't run on this branch yet (a maintainer needs to approve the workflow run), and main requires signed commits, so the commits here will need signing. The motion-blur background issue you listed under out of scope looks like a real bug and is probably worth its own issue.

Verdict: COMMENT Reasoning: No code blockers found and the touched tests pass locally, but the required CI suite has not run at this head and the commits are unsigned, so it is not ready to approve yet.

Thanks for the thorough review!

  1. validateDistributedRenderConfig now rejects chromaSubsampling other than "420"
    (tests added for both the Lambda and Cloud Run SDKs), and the rendering guide says
    distributed renders encode 4:2:0 only.
  2. Added chromaSubsampling.integration.test.ts: renders a small fixture with "444" and
    with the default and checks pix_fmt with ffprobe (yuv444p / yuv420p). I confirmed it
    fails if the option is dropped from the getEncoderPreset call.

I also rebased onto main and signed all commits, and moved the 444 drawElement gate into a
helper to keep runCompileStage under the fallow complexity threshold.

I'll open a separate issue for the motion-blur background bug.

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 7b1241d. Thanks for the quick turnaround, and for signing the commits.

Both points from my last review are handled:

  1. validateDistributedRenderConfig now rejects "444", and the rendering guide says so plainly. I checked that nothing returns early before the new check. I also confirmed that renderToLambda and renderToCloudRun call it first, so the plain JavaScript case is covered. With the check disabled, the new Lambda and Cloud Run test cases fail.
  2. The new integration test renders through executeRenderJob and checks pix_fmt. I ran it on Linux and it passes. Dropping the option from the getEncoderPreset call makes it fail with yuv420p, as you described.

The rebase onto main looks clean. The earlier commits only changed where they sit next to the new pageMotionBlur lines. The drawElement helper behaves the same as the inline version, and compileStage.test.ts still catches it if the gate is skipped. The encoder side also still holds: GPU encoders are only reached with useGpu, which every entry point refuses for "444". No h264 profile is forced, so libx264 picks High 4:4:4 Predictive.

One small follow-up, not blocking: the new test only covers the encoder half. If chromaSubsampling is dropped from the resolveCaptureSurface call in the orchestrator (or in the probe stage too), frames are captured as JPEG and encoded as yuv444p. The file reports yuv444p but has 4:2:0 colour, and the test still passes; I checked by making that change. Asserting the capture side as well would close the gap. You could check that the render captured PNG, or compare chroma detail at the edges of the red text between the two outputs.

Before merge: the repository's CI still hasn't run on this branch. A maintainer needs to approve the workflow run so the unit and integration lanes run the new test.

Verdict: COMMENT
Reasoning: Both earlier points are addressed and verified, and I found no blockers, but CI has not run on this head yet, so it is not ready to approve.

@jrusso1020

Copy link
Copy Markdown
Collaborator

Yes as an opt-in. Please rebase and get the reachability and captures checks green.

@PaprikaChen
PaprikaChen force-pushed the feat/chroma-subsampling-444 branch from 1d03558 to 2e4583a Compare October 26, 2026 08:44
@PaprikaChen

Copy link
Copy Markdown
Contributor Author

Rebased onto main. Test reachability is re-pinned, and the description now has a "No visible change" section for the captures check. Could someone approve the workflow run?

@miguel-heygen: added the capture-side check (the test now fails if chromaSubsampling is dropped from resolveCaptureSurface), and switched to transparent / capturesAlpha from #6262.

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 2e4583a. Thanks for closing the capture gap so precisely.

The point from my last review is fixed. The new edge check puts the red/blue boundary inside a 4:2:0 chroma pair. That makes it a direct test that the frames were captured at full chroma resolution. I ran it on Linux:

  • It passes three times in a row.
  • With chromaSubsampling dropped from the capture surface in the orchestrator and the probe stage, it fails (Cb difference 1), while the pix_fmt check still passes. That was exactly the gap.
  • With the option dropped from getEncoderPreset, both 4:4:4 checks fail.
  • With the capture surface forced to clear the background, the edge check fails too, so the opaque background is covered end to end.

The switch to transparent / capturesAlpha from #6262 looks right. resolveCaptureSurface sets transparent: needsAlpha, the same value main sets on both the probe and the main capture options. Alpha output, motion blur and plain renders therefore behave exactly as on main, and 4:4:4 gets opaque PNG. No transparentBackground references are left. The renamed verification-page test pins the opaque case. The opaque-PNG drawElement gate now also covers motion-blur MP4, as the description says. That fits, since sub-frame motion blur already requires screenshot capture.

The comment trims only removed or shortened comments. The comment ratchet and citation checks pass locally. Engine and producer typecheck cleanly, and test reachability and the producer test classification check pass.

Before merge: the repository workflows have not run on this head yet (they are waiting for a maintainer to approve the fork run), including the reachability and PR captures checks. I'd approve once they are green.

Verdict: COMMENT
Reasoning: Every earlier point is fixed and verified, and I found no new issues, but CI has not run on this head yet, so it is not ready to approve.

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fork workflows have now run on this head and every check is green. Everything else is as in my last review: the earlier point is fixed, and the switch to transparent / capturesAlpha matches main. Whether this feature goes in is for the maintainers to decide.

Verdict: APPROVE
Reasoning: Every earlier point is fixed and verified, and all checks pass at this head.

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One change before this merges: three comment blocks are longer than the repository allows (at most 3 to 4 lines, saying what the code cannot):

  • packages/producer/src/services/render/chromaSubsampling.integration.test.ts:1 (5 lines)
  • packages/producer/src/services/render/chromaSubsampling.ts:1 (5 lines)
  • packages/producer/src/services/renderOrchestrator.ts:416 (8 lines)

Please cut each to one to three plain lines and keep the reasoning in the PR description, which already carries the measurements. Everything else is ready: I ran the chroma tests across producer, engine and CLI (all pass) and, with the PR's sources put back to the merge base, the 4:4:4 cases fail, so they guard the change. Once the trimmed head is green it goes straight into the merge queue.

@PaprikaChen
PaprikaChen force-pushed the feat/chroma-subsampling-444 branch 3 times, most recently from 8bf20df to cb4ddb0 Compare October 28, 2026 07:43
@PaprikaChen

Copy link
Copy Markdown
Contributor Author

Trimmed the comment blocks and rebased onto main. Could someone approve the workflow run?

@PaprikaChen

Copy link
Copy Markdown
Contributor Author

@miguel-heygen the comment blocks are trimmed as requested. main moves quickly, so this keeps picking up conflicts. Happy to rebase right away whenever you're ready to approve the workflow run, just let me know.

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at cb4ddb090de651f8daeb69025e5a1cae7bef3305.

The code is clean at this head. Miguel's latest request to trim the three comment blocks is resolved. The earlier cloud-validator and capture-test gaps are also fixed: unsupported cloud requests fail explicitly, and the integration test checks full-resolution captured colour as well as the output pixel format.

The head conflicts with current main in CLI plan.ts, producer integration-tests.json, compileStage.ts and its test, renderOrchestrator.ts, Studio server encoderOptions.ts/types.ts, and skills-manifest.json. Rebase while preserving the chroma option through both capture and encoding and retaining the newer shared parser fields. Regenerate the manifest and rerun the relevant tests; the resulting head needs review before approval.

Validation at this SHA: full build passed; 630 focused tests passed across engine, CLI, producer, Studio and cloud validators. Three real Chrome/FFmpeg integration checks passed: yuv444p, full-resolution chroma at the opaque colour edge, and default yuv420p. Removing 444 from capture-format resolution makes the edge test fail while the pixel-format test still passes (1 failed, 2 passed); restored afterward.

Audited: PNG/background resolution for probe and main capture, drawElement gates, preset propagation across encode routes, resume identity, CLI/Docker/request/server forwarding, unsupported-format/GPU/HDR/cloud guards, and existing reviews. Trusting: untouched render interiors and the author's player-compatibility measurements. HDR/alpha compatibility was traced and covered by unit controls, without a new end-to-end replay. Independent adversarial pass found no additional blocker.

Need: yes, opt-in full-resolution colour masters · Size: ok, one contract across entry points · Reuse: existing encoder preset and capture surface · Simpler: already minimal · Structure: fine
Risk: medium — changes capture and encoder contracts; unsupported routes fail explicitly.

— hf-review

Verdict: COMMENT
Reasoning: Code merits and prior fixes are clean, but this conflicting head needs a rebase and review of the resulting revision before approval.

@PaprikaChen
PaprikaChen force-pushed the feat/chroma-subsampling-444 branch from cb4ddb0 to 9d420b5 Compare November 7, 2026 01:03
@PaprikaChen

Copy link
Copy Markdown
Contributor Author

Reviewed at cb4ddb090de651f8daeb69025e5a1cae7bef3305.

The code is clean at this head. Miguel's latest request to trim the three comment blocks is resolved. The earlier cloud-validator and capture-test gaps are also fixed: unsupported cloud requests fail explicitly, and the integration test checks full-resolution captured colour as well as the output pixel format.

The head conflicts with current main in CLI plan.ts, producer integration-tests.json, compileStage.ts and its test, renderOrchestrator.ts, Studio server encoderOptions.ts/types.ts, and skills-manifest.json. Rebase while preserving the chroma option through both capture and encoding and retaining the newer shared parser fields. Regenerate the manifest and rerun the relevant tests; the resulting head needs review before approval.

Validation at this SHA: full build passed; 630 focused tests passed across engine, CLI, producer, Studio and cloud validators. Three real Chrome/FFmpeg integration checks passed: yuv444p, full-resolution chroma at the opaque colour edge, and default yuv420p. Removing 444 from capture-format resolution makes the edge test fail while the pixel-format test still passes (1 failed, 2 passed); restored afterward.

Audited: PNG/background resolution for probe and main capture, drawElement gates, preset propagation across encode routes, resume identity, CLI/Docker/request/server forwarding, unsupported-format/GPU/HDR/cloud guards, and existing reviews. Trusting: untouched render interiors and the author's player-compatibility measurements. HDR/alpha compatibility was traced and covered by unit controls, without a new end-to-end replay. Independent adversarial pass found no additional blocker.

Need: yes, opt-in full-resolution colour masters · Size: ok, one contract across entry points · Reuse: existing encoder preset and capture surface · Simpler: already minimal · Structure: fine Risk: medium — changes capture and encoder contracts; unsupported routes fail explicitly.

— hf-review

Verdict: COMMENT Reasoning: Code merits and prior fixes are clean, but this conflicting head needs a rebase and review of the resulting revision before approval.

Rebased onto latest main. Conflicts resolved keeping main's new fields (ProRes profile, video-layer capture overrides, template entry resolution) alongside the chroma option, manifest regenerated. Locally: build, typecheck, comment-ratchet/reachability, the touched unit tests and the three chroma/motion-blur/first-frame integration tests all pass. Could you approve the workflow run and take another look?

@PaprikaChen

Copy link
Copy Markdown
Contributor Author

main moved again after the rebase; the only conflict is the generated skills-manifest.json and a docs page, not the chroma code. I'll rebase right before merge whenever you're ready to approve the run.

PaprikaChen and others added 7 commits November 10, 2026 15:24
PNG capture used to imply a cleared page background. A new
CaptureOptions.transparentBackground lets lossless opaque PNG capture
(4:4:4 MP4 output) keep the page background. Absent, it still follows
format === "png", so alpha output and motion blur are unchanged.

Opaque PNG skips drawElement, whose verification and worker-encode
paths are JPEG-only.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
getEncoderPreset takes an optional chromaSubsampling ("420" by
default). "444" returns yuv444p for SDR MP4, so libx264 encodes the
High 4:4:4 Predictive profile and odd sizes need no even padding.
WebM, MOV and HDR presets ignore it. The ChromaSubsampling type is
exported for the producer.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ture

RenderConfig.chromaSubsampling ("420" by default) opts an SDR MP4
render into 4:4:4. "444" encodes libx264 yuv444p and captures every
frame as opaque PNG: Chrome's quality-100 JPEG is already 4:2:0 on
beginFrame and screenshot capture, and drawElement can fall back to
screenshot mid-render, so PNG is the only capture that keeps the
colour. drawElement is turned off for these renders.

The probe and main capture options share resolveCaptureSurface, so
they cannot disagree on format, quality or background.

Formats other than mp4, GPU encoding and HDR output are rejected
before any work starts, auto-detected HDR sources fail before capture,
and distributed requests refuse 444. The default render is unchanged.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared parseEncoderOptions validates chromaSubsampling and answers
400 for "444" with a non-mp4 format, gpu, or force-hdr, so a request
that would come back 4:2:0 never starts a render. The Studio render
route, both Studio adapters and the producer HTTP server forward it.
The producer server also refuses "444" with outputDynamicRange "hdr",
which parseEncoderOptions cannot see.

No Studio UI control; the option is route-level like crf and hdrMode.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`hyperframes render --chroma-subsampling 444` opts an SDR MP4 render
into full-resolution chroma (yuv444p with lossless PNG capture). The
default stays 420. The plan rejects 444 with a non-mp4 format, --gpu
or --hdr before anything renders, with a usage error that names the
conflicting flag.

The value reaches the local render request and, only when it is 444,
the Docker argv, so default Docker invocations are unchanged.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a "4:4:4 masters" section to the rendering guide: when to use
--chroma-subsampling 444, why it captures PNG, the measured chroma gain
(large for text and UI, none for footage and gradients), its limits,
and a player compatibility table to be filled in from testing.

Document the flag in the CLI reference, the built-in rendering docs and
the hyperframes-cli skill reference (with the regenerated skills
manifest), and the chromaSubsampling option in the producer docs.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fill the 4:4:4 compatibility table from testing H.264 High 4:4:4
Predictive output. Chrome 155, Edge 155, VLC 3.0, Safari on iOS 26.6
and Safari 26.6 on macOS play it, and so does AVFoundation on macOS
26.6, the framework QuickTime Player plays through. Firefox 157 on
Windows and both Windows media players do not.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PaprikaChen and others added 12 commits November 10, 2026 15:24
Replace the vague "slower capture" note with measured numbers: a text
and UI composition rendered in about twice the default time, full-frame
footage in about three and a half times, because PNG encoding cost grows
with image detail.

Refs heygen-com#5555

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Distributed configs have no chroma field and validateDistributedRenderConfig
ignored unknown keys, so a JS caller passing chromaSubsampling: "444" to the
Lambda or Cloud Run SDK got a 4:2:0 file with no error. Reject it, and make
the rendering guide say so precisely.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every helper had tests, but nothing checked the orchestrator passes
chromaSubsampling through, so dropping it from the encoder preset or capture
surface would turn 444 renders back into 4:2:0 with all tests green. Render a
small fixture with "444" and with the default and check pix_fmt with ffprobe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keeps runCompileStage under the fallow complexity threshold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
comment-ratchet failed on 11 files this PR touched. Keep the 4:4:4
rationale once in chromaSubsampling.ts, move the distributed-render
reason into its error message, and cut the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The encoder check alone passes when chromaSubsampling is dropped from the
capture surface: frames are captured as JPEG and encoded yuv444p with
4:2:0 colour. Put a red/blue edge at an odd x, inside a 4:2:0 chroma
pair, and require the two pixels to differ in Cb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…comment share

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The merge that adopted main's transparent / capturesAlpha left these files
uncommitted, so resolveCaptureSurface still returned transparentBackground.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main added CaptureOptions.transparent and capturesAlpha (heygen-com#6262), which do
what this PR's transparentBackground did. Use them, keep the PNG check in
drawElementWorkerEncodeEligible, and re-pin test-classification.mjs and
the skills manifest against current main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reasoning lives in the PR description with the measurements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep main's video-layer capture overrides, template entry resolution,
ProRes profile parsing and blur-differential tests alongside the chroma
option, and fold the encoder sub-parsers into one loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PaprikaChen
PaprikaChen force-pushed the feat/chroma-subsampling-444 branch from 9d420b5 to bdd3399 Compare November 10, 2026 23:25

@miguel-heygen miguel-heygen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at bdd339975a6f6c3a127c9f6a23f14a88299e463c. This approval replaces my request for changes at 2e4583ad.

The three comment trims are in

  • chromaSubsampling.integration.test.ts:1 is now 2 lines.
  • chromaSubsampling.ts:1 is now 1 line.
  • The chromaSubsampling field on RenderConfig (renderOrchestrator.ts:455, which was :416 before the rebase) is now 1 line.

The comment checks pass locally against the merge base 7ff8ece46: citations are OK (35 files examined), and the comment ratchet reports all 17 changed source files at or under their share.

What changed since cb4ddb09 (jrusso1020's clean pass)
I range-diffed the 17 commits from cb4ddb09's base against the 19 at this head and compared the net PR diff file by file.

  • 36 of the 40 files carry exactly the same changed lines as at cb4ddb09. That includes every chroma source file:
    • chromaSubsampling.ts
    • captureImageFormat.ts
    • compileStage.ts
    • probeStage.ts
    • chunkEncoder.ts
    • frameCapture.ts
    • server.ts
    • renderRequest.ts
    • renderConfigValidation.ts
    • the CLI plan, render and docker files
    • the docs
  • The rebase was not purely mechanical. 1538c0de9 ("reconcile 4:4:4 chroma with latest main") adds main's later work back alongside the chroma option. Its net effect on the PR is limited to four files:
    • renderOrchestrator.ts capture options: main's video-layer passthrough (added after cb4ddb09 forked) now goes after ...captureSurface as format: "png", quality: undefined, transparent: true, hiddenElementIds. This matches main's behaviour.
      • Main passed a JPEG quality with format: "png" here, and this head passes undefined. The screenshot path is the only one the video layer uses (it forces screenshot), and it drops quality for PNG anyway (screenshotService.ts:159/250), so the output is the same.
      • With 444 plus a video layer, the capture stays PNG, and the video layer's transparent overlay wins. That is correct for compositing over the passthrough video.
    • studio-server encoderOptions.ts: main's parseProres and the chroma parser are folded into one loop. It keeps the same order (rate, delivery, prores, chroma), returns the first error, and merges options as before.
    • chunkEncoder.test.ts: formatting only.
    • skills-manifest.json: the hash was regenerated.
  • bdd339975 adds back main's resolveEntryHtmlPath tests that the rebase had dropped. It is test-only, and compileStage.ts is identical to main apart from the PR's own gate.

The tests still guard the change

  • All chroma suites pass locally:
    • engine chunkEncoder and frameCapture-verificationBackground: 116
    • producer compileStage, renderRequest and server.encoderOptions: 136
    • producer chromaSubsampling and captureImageFormat (bun): 17
    • studio-server encoderOptions and routes/render: 116
    • aws-lambda (40) and gcp-cloud-run (20) validateConfig
  • I put each PR source back to the merge base, one at a time. Each time its tests fail:
    • chunkEncoder.ts: 1
    • captureImageFormat.ts: the suite fails
    • encoderOptions.ts: 7
    • compileStage.ts: 1
    • renderRequest.ts: 2
    • renderConfigValidation.ts: 1
  • Typecheck is clean for engine, producer and studio-server. The CLI's only typecheck errors are an unbuilt @hyperframes/inspector/live in my checkout, nothing in the chroma files. Three CLI render.test.ts cases fail locally only because a spawned worker can't find @hyperframes/core/dist without a build.

CI
The fork's workflows have not run at this head. CI, regression, Comments, Docs, PR captures, Windows render verification, CodeQL, preview-regression and Player perf all show action_required, waiting for a maintainer to approve the run. None of the required checks exist yet, so a maintainer needs to approve the workflow runs, and the required checks will still gate the merge.

Verdict: APPROVE
Reasoning: The one thing I asked for (the comment trims) is done. Since the last clean review, the only changes are rebase reconciliation that keeps main's video-layer and ProRes behaviour, and every chroma file is unchanged. The chroma tests pass and fail when the sources are reverted. The required checks still have to run once a maintainer approves the fork workflows.

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at bdd339975a6f6c3a127c9f6a23f14a88299e463c, following my review of cb4ddb090de651f8daeb69025e5a1cae7bef3305.

The prior integration concerns are resolved. Range-diff and net patch comparison show identical added/removed lines in 36 of 40 files. I read the remaining reconciliation: main's video-layer PNG/transparent/hidden-element override remains after the chroma capture surface; the encoder parser loop preserves rate/delivery/ProRes options and first-error precedence; the chunk-encoder test change is formatting; and the skills manifest is regenerated. Main's canonical entry-resolution implementation and tests are retained. The targeted production reconciliation was already present at 9d420b5c; the final commit restores the entry tests and updates the manifest.

The opt-in still reaches both probe/main PNG capture and the encoder's yuv444p preset. Default and explicit 420 preserve the old path. The three requested comment trims remain in place. Independent adversarial delta inspection found no blocker.

Validation at this head: pytest-gate.sh bun test packages/producer/src/services/render/chromaSubsampling.test.ts reports 7 pass / 0 fail. A direct parser probe reports 32 base/current parity cases plus 5 chroma controls passed. bun packages/cli/scripts/gen-skills-manifest.ts --check passes. The required GitHub checks pass; the CI run's head SHA was verified as this exact commit. I did not repeat the earlier full build or real Chrome/FFmpeg integration render, so the earlier 630-test and render measurements are not fresh validation of this head.

Audited: reconciliation delta, retained entry-resolution tests, capture/encoder forwarding and current main's video-layer handling. Trusting: unchanged render interiors and historical player compatibility measurements. No new split: the merged parser uses the existing pure sub-parsers, and the render uses the existing capture-surface helper.

Need: yes, opt-in full-resolution colour masters · Size: cohesive option across entry points · Reuse: existing capture surface and encoder sub-parsers · Simpler: already minimal · Structure: fine (checked resolveCaptureSurface, parseProres and capture overrides; no split)
Risk: medium - capture and encoder contracts with explicit unsupported-route guards.

Verdict: APPROVE
Reasoning: The rebased head preserves the prior chroma change and current main's rendering behavior, including the conflict-resolution paths that required a new review. No blocking defect found in the delta.

Reviewed by hf-review.

@jrusso1020
jrusso1020 added this pull request to the merge queue Nov 11, 2026
Merged via the queue into heygen-com:main with commit 1a9fb8a Nov 11, 2026
95 checks passed
@miguel-heygen miguel-heygen mentioned this pull request Nov 11, 2026
3 of 4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Opt-in 4:4:4 chroma for MP4 masters (measure first)

3 participants