Realtime WebRTC: no way to pin the ICE UDP port, which makes firewalled deployments impractical

[jimmykarily]

Is your feature request related to a problem? Please describe.

/v1/realtime WebRTC gathers its host ICE candidate on a random ephemeral UDP port, a different one on every call. There is no setting to pin it, so an operator running LocalAI on a host with a firewall cannot write a rule for it. The only rule that works is "allow all inbound UDP", which is a hard sell on a shared or remote machine.

The setup where this shows up: LocalAI on one machine on the LAN, a browser client on another machine connecting to /v1/realtime for streaming audio. Signaling is HTTP on the API port, so it succeeds and returns a valid answer. Media never flows, and the browser's peer connection goes connecting -> failed -> retry, indefinitely.

Posting the same offer to /v1/realtime/calls three times returns three different ports:

a=candidate:... 1 udp 2130706431 192.168.1.73 49743 typ host
a=candidate:... 1 udp 2130706431 192.168.1.73 37785 typ host
a=candidate:... 1 udp 2130706431 192.168.1.73 33836 typ host

The advertised address is correct, so LOCALAI_WEBRTC_NAT_1TO1_IPS is not the missing piece. The host firewall (firewalld's default REJECT ... icmp-host-prohibited) rejects UDP to those ports while the API port stays open. A connected UDP socket to any of them returns EHOSTUNREACH, while TCP to the API port connects normally.

webRTCSettingEngine (core/http/endpoints/openai/realtime_webrtc_ice.go:17) currently configures only SetNAT1To1IPs and SetInterfaceFilter. Both control which address is advertised. Neither controls which port, so pion uses its default random-port behavior.

Describe the solution you'd like

An opt-in setting to pin the UDP port, alongside the two existing WebRTC knobs in that same function.

Preferred: a single multiplexed port via pion's SetICEUDPMux.

LOCALAI_WEBRTC_UDP_PORT=8080

if cfg.WebRTCUDPPort > 0 {
    conn, err := net.ListenUDP("udp", &net.UDPAddr{Port: cfg.WebRTCUDPPort})
    // ...
    s.SetICEUDPMux(webrtc.NewICEUDPMux(nil, conn))
}

One port serves all concurrent sessions, which reduces the firewall and port-forward requirement to a single documented line ("open TCP <api> and UDP <port>"). This is pion's own stated purpose for the API:

SetICEUDPMux allows ICE traffic to come through a single UDP port, drastically simplifying deployments where ports will need to be opened/forwarded.

(pion/webrtc/v4, settingengine.go)

Unset by default, so existing behavior does not change.

A range option (LOCALAI_WEBRTC_UDP_PORT_RANGE=50000-50100, mapping to SetEphemeralUDPPortRange) would also solve it and could be offered alongside, but the mux is the better default since it is one firewall rule instead of a hundred. Other WebRTC servers expose one or both of these for the same reason, for example LiveKit's rtc.udp_port (single mux port) and Janus's rtp_port_range (a range).

pion/webrtc/v4 v4.2.11 is already a direct dependency, so both APIs are available with no new dependencies.

Describe alternatives you've considered

• LOCALAI_WEBRTC_NAT_1TO1_IPS and LOCALAI_WEBRTC_ICE_INTERFACES: these control which address is advertised, not which port is bound, so neither helps here.
• Allowing all inbound UDP on the host: works, but is not reasonable guidance to give users, and is often not permitted.
• A TURN relay: heavy infrastructure for a plain LAN deployment.

Additional context

Version: v4.7.1-200-g4b631fa3.

Related observation: when ICE fails this way, LocalAI logs nothing at all. The session is created, the answer is returned, and then it goes quiet. A log line when a /v1/realtime peer connection reaches failed, or fails to reach connected within a timeout, would make this much faster to diagnose. From the client side it is indistinguishable from a model or pipeline problem. Happy to file that separately if preferred.

I am willing to open a PR for the mux option if the approach sounds right.

---

Investigated and drafted with AI assistance. Reviewed, verified and submitted by a human.

[localai-org-maint-bot]

I’m picking this up. I’ll add an opt-in fixed UDP mux port for Realtime WebRTC, wire it through configuration and documentation, and cover both configured and default behavior with focused tests on latest master.