Support scaled cameras (WebXR rig scaling) - #403
Open
marwie wants to merge 1 commit into
Open
Conversation
…D thresholds, XR camera choice
Gaussian splat scenes break under a uniformly scaled camera — the common
WebXR pattern where an engine scales the user rig to let people resize a
placed scene in AR/VR:
1. Rendering: the accum-to-view transform is decomposed into position +
quaternion, discarding the scale. The translation keeps the scale
factor but the rotated center term and the splat extents don't, so the
scene warps instead of scaling. New renderToViewScale uniform (exact
for uniform scale: T*R*S decompose => view = s*R*center + t); the
covariance branch already carries the full basis and is unchanged.
2. Sort/LOD triggers: the view-changed epsilon (1mm) and the LOD update
ramp (1m) are world-unit constants. At rig scale s, physical head
movement maps to s-times less/more world movement — scaled-up scenes
never re-sort ('frozen' blend order, foveation holes). Both now scale
with the camera's world scale. The LOD pixel cut is deliberately NOT
scaled: perspective projection cancels a uniform camera scale for
size-at-distance; the apparent-size change comes entirely from the
world-space distances the cut already measures.
3. XR update camera: updates ran on the ArrayCamera container, often in a
setTimeout outside the render pass — where the parentless container's
getWorldPosition recomputes matrixWorld from the raw reference-space
local matrix, discarding the rig transform entirely. Use the per-eye
camera instead (valid at any time when the engine keeps it parented;
half an IPD off head center).
Unscaled cameras: behavior is bit-identical (scale factor 1 everywhere).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Splat scenes break under a uniformly scaled camera. In Needle Engine we scale the user's XR rig — not the scene — when users resize a placed scene in AR/VR: world coordinates stay stable, which keeps networked multi-user sessions consistent (every participant shares the same scene space while viewing at their own scale). Under such a scaled camera, three issues stack up in Spark; all fixed here, and with an unscaled camera every change is a no-op (scale factor 1 throughout).
Render transform drops the camera scale.
The accum→view transform is decomposed to pos + quat; the translation keeps the scale factor but the rotated centers and splat extents don't → the scene warps instead of scaling. Fix:
renderToViewScaleuniform from the previously-discarded decompose scale (scales *= s,viewCenter = s * quatVec(q, center) + pos). TheenableCovSplatsbranch already uses the full basis and is unchanged;OldSparkRendereruntouched.Sort/LOD triggers are world-unit constants.
The view-changed epsilon (1 mm) and LOD ramp (1 m) don't account for camera scale: at rig scale s, physical head movement maps to s× less world movement — a scene scaled up 20× never re-sorts (frozen blend order, foveation holes). Fix: both thresholds scale with the camera's world scale. The LOD pixel cut is deliberately not scaled — perspective cancels uniform camera scale for size-at-distance (device-verified: scaling it double-counts).
XR updates read the ArrayCamera container outside the frame.
updateInternalruns onrenderer.xr.getCamera(), often in asetTimeout: the parentless container'sgetWorldPosition()recomputesmatrixWorldfrom the raw reference-space local matrix, discarding the rig transform — rendering is correct, butsortedCenter/viewToWorldlive in raw head space. Fix: use the per-eye camera (getCamera().cameras[0] ?? getCamera()), which engines can keep validly composed at all times.Found and device-verified building AR splat viewing on Needle Engine (Pixel 9, Chrome
immersive-ar): pinch-scaling a placed splat scene now scales correctly and keeps sorting/LOD alive at any rig scale.Demo scene: https://splat-z23hmxbhac3d-zjprnz.needle.run/ (Enter AR and scale the scene with two fingers)