Recording/detection may not recover after go2rtc stream enters STUCK state | 0.42.4

[rizkybs70]

go2rtc stream fails to recover after STUCK state, causing recording and detection gaps

Description

I have 2 CCTV sites in different cities connected through a site-to-site VPN L2TP.

LightNVR is located at Site A, with several local cameras. There are also 2 remote cameras at Site B connected through the VPN.

All cameras initially work normally, including live view, recording, and object detection.

However, after LightNVR has been running for several hours, the remote cameras can experience long recording/detection gaps while the local cameras continue working normally.

Restarting LightNVR from System -> Restart LightNVR restores the affected streams temporarily.

I have reproduced the issue on LightNVR 0.41.4 until latest.

Topology

Network / Deployment Topology

                    ┌─────────────────────────────┐
                    │           Site A            │
                    │      LightNVR Location      │
                    └─────────────────────────────┘

        Local CCTV 1 ─────┐
        Local CCTV 2 ─────┤
        Local CCTV 3 ─────┤
        Local CCTV n ─────┤
                          │
                          ▼
                 ┌──────────────────┐
                 │   LightNVR Host  │
                 │                  │
                 │  Docker Host     │
                 │                  │
                 │  ┌────────────┐  │
                 │  │ LightNVR   │  │
                 │  │            │  │
                 │  │  go2rtc    │  │
                 │  │  Recorder  │  │
                 │  │  Detection │  │
                 │  └────────────┘  │
                 │                  │
                 │  ┌────────────┐  │
                 │  │ SSD        │  │
                 │  │ MobileNet  │  │
                 │  │ v2 / LOD   │  │
                 │  └────────────┘  │
                 └────────┬─────────┘
                          │
                          │
                          │ Site-to-Site VPN
                          │
                          ▼

                    ┌─────────────────────────────┐
                    │           Site B            │
                    │      Remote CCTV Site       │
                    └─────────────────────────────┘

                      ┌──────────────────┐
                      │ Remote Camera 1  │
                      │ `Kamar`          │
                      └──────────────────┘

                      ┌──────────────────┐
                      │ Remote Camera 2  │
                      │ `Teras_Kos`      │
                      └──────────────────┘

Data Flow

For local cameras:

Local Camera
    ↓
LAN
    ↓
LightNVR / go2rtc
    ↓
Recording + Object Detection

For remote cameras:

Remote Camera
    ↓
Remote LAN
    ↓
Site-to-Site VPN
    ↓
LightNVR Host
    ↓
go2rtc
    ↓
rtsp://localhost:8554/<stream>?video
    ↓
MP4Writer + Object Detection

The issue only becomes visible on the remote-camera path.

Local cameras connected directly to Site A continue recording and producing detection events normally while the remote cameras may enter the failure state.

Environment

• Host : STB HG680P
• LightNVR running in Docker
• SSD MobileNet v2 detector running in Docker
• LightNVR and detector are on the same host
• 2 remote cameras connected through site-to-site VPN
• Several local cameras on the same LAN as LightNVR
• Object detection enabled on the affected remote cameras

Observed Behavior

The clearest example is the remote camera Kamar.

Before a significant recording gap, LightNVR started reporting:

[2026-09-23 08:33:08.024] [ERROR] [MP4Writer] [Kamar] Failed to record segment for stream Kamar (error: -1), implementing retry strategy...

[2026-09-23 08:34:10.670] [ERROR] [MP4Writer] [Kamar] Failed to record segment for stream Kamar (error: -1), implementing retry strategy...

[2026-09-23 08:34:35.583] [ERROR] [go2rtc] Stream Kamar: appears STUCK - no data flow for 3 checks

[2026-09-23 08:34:40.588] [ERROR] [go2rtc] CURL request failed: Timeout was reached

[2026-09-23 08:34:40.589] [ERROR] [go2rtc] Failed to unregister stream from go2rtc: Kamar

After that, both MP4Writer and Detection start failing to access the internal go2rtc stream:

[2026-09-23 08:36:06.417] [ERROR] [MP4Writer] [Kamar] Failed to open RTSP input rtsp://localhost:8554/Kamar?video: -875574520 (Server returned 404 Not Found)

[2026-09-23 08:36:06.417] [ERROR] [MP4Writer] [Kamar] Failed to record segment for stream Kamar (error: -875574520), implementing retry strategy...

[2026-09-23 08:36:09.533] [ERROR] [Detection] [Kamar] Failed to open input: Server returned 404 Not Found

The same pattern continues much later:

[2026-09-23 09:29:36.498] [ERROR] [Detection] [Kamar] Failed to open input: Server returned 404 Not Found

[2026-09-23 09:29:55.444] [ERROR] [go2rtc] CURL request failed: Timeout was reached

[2026-09-23 09:29:55.444] [ERROR] [go2rtc] Failed to unregister stream from go2rtc: Kamar

[2026-09-23 09:30:12.731] [ERROR] [MP4Writer] [Kamar] Failed to open RTSP input rtsp://localhost:8554/Kamar?video: -875574520 (Server returned 404 Not Found)

And it is still happening later:

[2026-09-23 11:16:25.217] [ERROR] [go2rtc] CURL request failed: Timeout was reached

[2026-09-23 11:16:25.218] [ERROR] [go2rtc] Failed to unregister stream from go2rtc: Kamar

[2026-09-23 11:16:30.502] [ERROR] [MP4Writer] [Kamar] Failed to open RTSP input rtsp://localhost:8554/Kamar?video: -875574520 (Server returned 404 Not Found)

[2026-09-23 11:16:33.629] [ERROR] [Detection] [Kamar] Failed to open input: Server returned 404 Not Found

The internal RTSP stream was still returning 404 around 12:31:

[2026-09-23 12:31:35.700] [ERROR] [MP4Writer] [Kamar] Failed to open RTSP input rtsp://localhost:8554/Kamar?video: -875574520 (Server returned 404 Not Found)

[2026-09-23 12:31:38.826] [ERROR] [Detection] [Kamar] Failed to open input: Server returned 404 Not Found

[2026-09-23 12:32:15.256] [ERROR] [Detection] [Kamar] Failed to open input: Server returned 404 Not Found

Based on the logs, the failure path appears to be approximately:

Remote RTSP stream stalls
        ↓
go2rtc detects STUCK stream
        ↓
LightNVR attempts to unregister it
        ↓
go2rtc API request times out
        ↓
internal go2rtc stream becomes unavailable
        ↓
rtsp://localhost:8554/<stream>?video returns 404
        ↓
MP4Writer and Detection continue retrying
        ↓
recording/detection becomes intermittent or stops for a long period

Camera Comparison

Kamar - Remote Site

Kamar shows significant recording gaps.

The logs for this camera clearly show the STUCK -> unregister timeout -> localhost RTSP 404 -> repeated retry pattern.

Teras_Kos - Remote Site

Teras_Kos, another camera at the same remote site, also stopped producing detection recordings for an extended period.

In my test it remained without new detection recordings until at least around 20:46.

I currently do not have equivalent stream-specific error logs for Teras_Kos, so I cannot confirm that its internal failure path is exactly the same as Kamar.

Teras - Local Site

Teras, located on the same site/LAN as the LightNVR server, continued producing detection events normally during the same period.

This suggests that the issue is not a complete failure of:

• LightNVR itself
• storage
• the object detector
• the recording subsystem globally

The issue appears to affect individual remote streams.

Network Evidence

The remote CCTV site is independently monitored using Uptime Kuma.

During the affected period:

• the remote site remained reachable
• there was no prolonged site-to-site VPN outage
• latency was generally low/stable
• there were occasional latency spikes
• there was no network outage corresponding to the many-hour recording/detection gap

I understand that this does not prove the RTSP session itself had zero packet loss, jitter, or TCP/session interruption.

A short RTSP or network interruption may still be what initially triggers the condition.

The main concern is that LightNVR/go2rtc does not appear to recover reliably afterwards.

Expected Behavior

If an RTSP stream temporarily stalls or disconnects, LightNVR should automatically recover when the source becomes available again.

The expected recovery would be roughly:

detect unhealthy stream
    ↓
reset/remove stale stream state
    ↓
re-register stream in go2rtc
    ↓
verify internal RTSP stream
    ↓
reconnect MP4Writer
    ↓
reconnect Detection
    ↓
resume normal recording/detection

A full LightNVR restart should not be required.

Actual Behavior

The affected stream can remain in a retry loop where both MP4Writer and Detection repeatedly receive:

Server returned 404 Not Found

from:

rtsp://localhost:8554/<stream>?video

The stream may partially recover or become intermittent, but recovery is not reliable.

Restarting LightNVR restores the affected cameras temporarily.

Possible Area to Investigate

Could this be related to the go2rtc recovery path after a stream is marked as STUCK?

In particular:

• What happens if unregistering a go2rtc stream times out?
• Is the stream guaranteed to be re-registered afterwards?
• Does LightNVR verify that the stream still exists in go2rtc?
• Could repeated 404 Not Found from localhost:8554/<stream> trigger a full stream recreation instead of only retrying MP4Writer/Detection?
• Could LightNVR reset only the affected stream instead of requiring a restart of the whole application?

It may also be useful to add error handling where multiple consecutive internal RTSP 404 responses cause LightNVR to check the go2rtc stream state and recreate the stream if necessary.

Workaround

Current workaround:

System -> Restart LightNVR

After restarting LightNVR, recording and object detection start working again.

The problem can return after LightNVR has been running for several hours.

Evidence / Attachments

I will attach the following evidence:

1. kamar-remote-timeline.png
   • Remote camera
   • Shows significant recording gaps

2. teras-kos-remote-timeline.png
   • Second remote camera
   • Shows detection recordings stopping for an extended period

3. teras-local-timeline.png
   • Local camera
   • Shows detection continuing normally during the same period

4. site-to-site-uptime-kuma-24h.png
   • Shows the remote site remained reachable during the recording/detection outage

5. lightnvr-error-log.txtlightnvr-error-log20260923-135508.txt
   • Full LightNVR log containing the STUCK, go2rtc timeout, MP4Writer, Detection, and internal RTSP 404 errors

Additional evidence I can provide if required:

• go2rtc API stream state while the issue is occurring
• Docker configuration
• LightNVR image/tag information
• detector container information
• camera codec/resolution/FPS/bitrate/GOP settings
• direct RTSP test while LightNVR is in the failed state
• additional debug logs

Private information such as RTSP credentials will be redacted.

[matteius]

Thank you for an unusually complete report; the timestamps made this tractable. Your error code -875574520 is FFmpeg's AVERROR_HTTP_NOT_FOUND, i.e. go2rtc answered the local RTSP DESCRIBE with 404, which it does whenever it cannot produce media for the stream right then.

What we found on our side:

• The DELETE that follows "appears STUCK" used a 5 s timeout, but the bundled go2rtc rewrites its config file before answering. On your box under recording I/O that took longer, so the delete was applied but LightNVR treated it as failed (your "Timeout was reached" lands exactly 5.005 s after STUCK) and then PUT the stream back without verifying anything.
• The recovery cooldown only counted from a successful recovery, so a failing one was retried every 30 s tick; the same reload churned every 60 to 90 s for hours. Your hourly error pairs are the reloads whose DELETE happened to hit an I/O stall.
• A stream that vanished from go2rtc was never re-added, and recording-only streams had no way to signal the 404s.

On main: DELETE gets the longer timeout and is verified against /api/streams when it times out; re-registration is verified; the cooldown doubles after every recovery not followed by data flow (60 s up to 15 min); missing streams are re-added directly; the recorder reports local 404s to the monitor. Finally, if three recoveries in a row produce no data flow while the camera still accepts TCP connections from LightNVR, the monitor triggers the existing rate-limited go2rtc restart itself (the action that fixed it for you), at most once per 30 minutes per stream and never for a camera that is offline.

Please try the next release. If it recurs, the lines to look for are scoped recovery #N since last data flow, Detached go2rtc preload consumer, data flow restored after N scoped recoveries and escalating to a go2rtc process restart. The go2rtc API stream state you offered (GET /go2rtc/api/streams while the stream is in the failed state) would still help pin down what go2rtc's producer is doing.

[rizkybs70]

Thank you very much for the detailed investigation and for implementing the fix so quickly.

I really appreciate how thoroughly you traced the issue and improved the recovery logic around go2rtc. The explanation makes a lot of sense and matches the behavior I observed in the field.

Great work on the fix, and thanks again for taking the report seriously and turning it into a proper recovery improvement.

I will wait for the next Docker image/release, test it again on the same setup, and monitor the remote cameras for a while.

If the issue is no longer reproducible and the streams recover properly without a manual LightNVR restart, I will close this ticket as fixed.

Keep up the great work, and thanks again!

[rizkybs70]

Unfortunately, the issue still persists on my environment after updating to LightNVR 0.42.8. @matteius

The behavior is still the same as before:

• the remote/off-site cameras eventually stop producing AI detection recordings
• the local/on-site cameras continue recording and detecting normally
• the remote camera live view is still available
• restarting LightNVR restores the affected remote detection recordings temporarily

I have attached updated screenshots showing the same behavior on 0.42.8.

So at least in my environment, the issue does not appear to be fully resolved yet.

I can collect another set of logs from the failed state if that would help with further investigation.