feat(linux): get _exact_ 4:4:4 via supersampling of 4:2:0 - #5570
Draft
alanjds wants to merge 2 commits into
Draft
Conversation
…scale AMD VCN and Intel Quick Sync have no 4:4:4 profile on any codec, so streaming at native resolution always subsamples chroma in hardware, causing visible fringing on small text. A known workaround is to stream at an integer multiple of the desktop resolution and let the client downscale: if the upscale is point (nearest-neighbour) sampling, every captured pixel becomes an identical NxN block, so the encoder's 4:2:0 chroma averaging reproduces the capture's own chroma exactly, prior to quantisation. Today the Linux GL capture path hardcodes GL_LINEAR for every texture (gl::tex_t::make()), so this property doesn't hold: the bilinear blit softens chroma before the encoder ever sees it, reducing fringing without eliminating it. Add chroma_supersample (default off). When enabled and the stream resolution is an exact, even integer multiple of the capture resolution in both axes, switch the source texture to GL_NEAREST for the capture-to-stream blit in sws_t, rather than changing tex_t::make() globally (which backs every texture in the GL path, including the cursor overlay). The factor must be even, not just integral: 4:2:0 chroma averages a 2x2 window of stream pixels, and that window only stays inside identical-valued blocks when the block size is even - measured an odd factor (3x) to be worse than the existing bilinear path, not just less exact, so odd factors fall back to bilinear with a logged warning. ConvertUV.frag's two-tap horizontal average is dropped to a single effective tap (width_i = 0) under point sampling, since both taps already resolve to the same texel once the source is piecewise-constant. With the option off, behavior is unchanged (same GL_LINEAR, same width_i). Verified in a numpy simulation of the GL sampling rules against a synthetic test pattern: pre-quantisation chroma error is exactly zero at 2x/4x, and worse than bilinear at 3x, confirming the even-factor restriction.
Local validation pass on the previous commit surfaced two gaps: - ConfigConsistencyTest.AllConfigOptionsExistInAllFiles failed: chroma_supersample was documented in configuration.md but never added to config.html's option defaults or en.json's locale strings, so the web UI had no way to set it. Add the option (ordered to match docs/config.html consistently, verified by ConfigOptionsInSameOrderWithinSections) and wire a Checkbox into Advanced.vue, gated to platform === 'linux' to match where the underlying code lives. - supersample_factor() was exposed in graphics.h as pure, testable logic but had zero test coverage. Add tests/unit/platform/linux/test_graphics.cpp covering the even-factor requirement (2x/4x/6x/8x/10x/12x accepted, 3x/5x/7x rejected - matching the numpy-verified exactness results from development), the >=2 minimum, non-integer ratios, anisotropic ratios, and non-positive capture dimensions. Also verified separately (not part of this diff): clang-format is clean on the changed lines, and a full BUILD_DOCS=ON doxygen pass produces zero warnings on any changed file (the one pre-existing doxygen error in this checkout is an unrelated alias-argument bug in the third-party/doxyconfig submodule's own example doc, reproducible on master).
|
alanjds
changed the title
alanjds
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
chroma_supersampleoption, that changes the scaling of a 1080p to 4k fromGL_LINEARtoGL_NEAREST.It makes the client-side downscale to be exactly like 4:4:4, not a very good approximate with a small chroma error.
Why
I tried to get rid of chroma aberrations on streaming by turning on 4:4:4. It worked, but I lost hardware acceleration on server and on client, because the GPUs simply cannot do it. (Strix Halo APU server, Intel client)
That lead to high CPU load on client, delays on server frames, decoding enqueueing, a mess.
After iterating, found the best compromise on my network to be spending a bit more bandwidth via streaming 4:2:0 1080p as 4k, then down to 1080p on client. This way everything runs on hardware and is almost identical to 4:4:4.
Digging on why the "almost", found that Sunshine uses a filter that added some blurry, and when the upscale is a X2 multiplication, the "just linear" produces no artifacts. So let's apply it instead!
What follows is just lore on the details.
Background
Streaming at an integer multiple of the desktop resolution and letting the client downscale is a workaround discussed elsewhere and that keeps both ends in hardware. See Moonlight issue #1671, "Recombined" 4:4:4 for the protocol-level version of the same idea (recombining supersampled 4:2:0 into 4:4:4 on the client)
However it here is cheaper to implement.
Caveat: Applies only on even integer multipliers
It only works good when the upscaled is even times of the original. Non-integer or odd times are better with the current implementation. But even multiples are better with GL_NEAREST
Tests:
Measurements
From my own setup (Bazzite/AMD Strix Halo host, VAAPI encode, KMS capture; Bazzite/Intel
UHD 620 client, Moonlight Qt,
DRI_PRIME=0, fullscreen 1080p), 1920x1080 desktop, 45fps,100 Mbps cap,
vaapi_rc = vbr, client fullscreen, steady state on a mostly-staticdesktop:
For reference, native 1080p 4:2:0 via VAAPI on the same host measured ~13 Mbps at a
40 Mbps cap. Looks like SW compresses better, ok, but the interesting here is that VAAPI does not gets that worst from 1080p to 4k. Worsen just about 10%