Add video-file playback to HTMLVideoElement (src) for VideoTexture

[julapy]

What

Adds video-file playback to HTMLVideoElement, so Babylon.js content can play a video file/URL and use its frames as a VideoTexture — the same web API, unmodified Babylon.js:

const video = new HTMLVideoElement();
video.src = "app:///assets/clip.mp4"; // or an absolute path / http(s) URL
video.loop = true;
video.muted = true;
const texture = new BABYLON.VideoTexture("vid", video, scene, false, true, undefined, { autoPlay: true, loop: true });
material.emissiveTexture = texture;
// ...standard control API: video.play(), pause(), currentTime = 3.0, video.duration, events, etc.

Previously the HTMLVideoElement polyfill only supported srcObject (a camera MediaStream). This adds the src path alongside it; the two sources are mutually exclusive, matching the web. Any number of videos can play simultaneously (each element owns its own decoder + texture).

How

Commit 1 — VideoPlayer backend:

• VideoPlayer.h: a small platform abstraction (play/pause/loop/muted/volume/seek, duration/currentTime/dimensions, per-frame UpdateTexture, and load/ended/seeked events).
• Apple/VideoPlayer.mm: AVPlayer + AVPlayerItemVideoOutput requesting BGRA frames (no YUV pass), viewed zero-copy via CVMetalTextureCache and blitted into a persistent MTLTexture, wired to Babylon's InternalTexture with the same bgfx::overrideInternal-on-render-thread retry pattern as CameraDevice::UpdateCameraTexture. Loops via seek-to-zero.

Commit 2 — HTMLVideoElement.src:

• Extends NativeVideo with src, currentTime (get/seek), duration, loop, muted, paused, ended, isNative, and events (loadedmetadata/loadeddata/canplay/playing/pause/seeked/ended/resize). Player callbacks (which may fire on any thread) are marshalled onto the JS thread via JsRuntime::Dispatch; the element self-references while it owns a source so it isn't collected mid-playback. UpdateTexture routes to the camera stream or the video player depending on the active source.

Only the Apple backend is implemented here; other platforms return nullptr from VideoPlayer::Create (video src is a no-op, srcObject/camera unaffected).

Testing

Verified on device (iPhone 15 Pro Max, iOS 26.5, Xcode 26, Babylon.js 9.9.1 UMD) via the iOS Playground: a 2000×2000 h264 clip on two independent HTMLVideoElements, both looping, one seeked out of phase mid-run to exercise the control API — sustained 60 FPS (~1 ms GPU). Also runs concurrently with an active immersive-ar session and WebXR raw-camera-access plane in the same scene.

🤖 Generated with Claude Code

[bkaradzic-microsoft]

Latest bgfx can do hardware video decoding on D3D11, D3D12, Vulkan, and Metal. You might want to check that feature so that feature it's not macOS only.

[julapy]

hi @bkaradzic-microsoft - "latest bgfx" - is that already in BN and is it just a matter of hooking up video decoding?

[ryantrem]

Yes, bgfx is the rendering abstraction (over D3D, OpenGL, Vulkan, Metal) that Babylon Native uses.

[bkaradzic-microsoft]

@julapy I need sync with latest and then HW video decoding will be available in BN.

[bkaradzic-microsoft]

@julapy I updated CMake to get latest bgfx which has HW video decoder support. You would need to enable BGFX_CONFIG_VIDEO:

BabylonNative/Dependencies/CMakeLists.txt

Line 71 in d45b303

target_compile_definitions(bgfx PRIVATE BGFX_CONFIG_VIDEO=0)

And there is example of video player:
https://github.com/BabylonJS/bgfx/blob/539e1febfba9c0ad85cee43266be9bb066225b3e/tools/texturev/video_player.cpp

[julapy]

@bkaradzic-microsoft thanks for syncing bgfx and pointing me at the gate + the texturev player — I went through video.h, video_mtl.cpp and the example to understand the shape of it.

If I've read it right: bgfx::VideoDecoder takes parameter sets + elementary-stream access units with PTS and hardware-decodes into a bgfx texture (VideoToolbox on Metal, D3D/Vulkan video elsewhere), and everything above the bitstream — container demux (l-smash in the example), pacing/clock, seek/loop — stays app-side. That's a really clean split, and it maps nicely onto this PR's structure: VideoPlayer.h is deliberately a small platform abstraction and the Apple AVPlayer file is just one implementation of it. A bgfx-video-based VideoPlayer looks like the natural way to fill in the platforms that currently return nullptr from VideoPlayer::Create — one implementation for D3D11/D3D12/Vulkan instead of per-platform media APIs, and it would land frames in a bgfx texture directly instead of the overrideInternal dance this PR borrows from the camera path.

Two questions before I go there, since HTMLVideoElement parity is the target:

1. Audio — I couldn't find an audio path in the decoder or the texturev player. Is audio intentionally out of scope for bgfx::VideoDecoder (i.e. apps bring their own audio decode + A/V sync clock), or is there something planned? Most content we render has audio baked into the video, which is a big part of why the Apple implementation sits on AVPlayer (demux, buffering, audio and sync come for free).

2. Network sources — the example reads local files through l-smash. For video.src = "https://…" semantics we'd need progressive download/buffered reads; is a custom l-smash reader the intended route, or would you expect apps to download-then-play?

Depending on the answers, my thinking would be: keep AVPlayer as the Apple backend (audio + streaming for free, zero added binary), and add a bgfx-video VideoPlayer for the other platforms in a follow-up PR — happy to take that on. Also glad to flip BGFX_CONFIG_VIDEO on in that follow-up rather than this one so this PR stays scoped.