Feat/optimize_long_link - #143
Open
wardseptember wants to merge 18 commits into
Open
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #143 +/- ##
=====================================================
+ Coverage 85.87630% 86.44751% +0.57120%
- Complexity 4330 4392 +62
=====================================================
Files 436 437 +1
Lines 14373 14632 +259
Branches 1287 1336 +49
=====================================================
+ Hits 12343 12649 +306
+ Misses 2030 1983 -47
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
一、总览(总)
本分支核心目标:把 tRPC/HTTP 客户端从"伪长连接 + 粗暴空闲扫描关闭"改造成真正可靠的长连接。
旧方案本质问题:框架基于 pull 模型维护服务端 IP,
RpcClusterClientManager用全局空闲扫描器不分协议按lastUsedNanos粗暴关闭客户端;Netty 客户端/服务端 Handler 又各自在IdleStateEvent里直接channel.close()。这套机制导致连接被无谓销毁、半死连接探测不到、请求被发到已断开/正在关闭的链接而报错,以及大量并发竞态。新方案按分层、分协议重构:Netty(tRPC)连接随
BackendConfig常驻、空闲治理下沉传输层、靠"懒重连 + closeFuture 回调"恢复;非 Netty(HTTP)靠连接池保活/驱逐 + 30s 集群扫描兜底;并系统性修复并发竞态、收敛默认参数。服务端保留一层可配置、可关闭的空闲兜底:idleTimeout到点关闭长时间收发均静默的连接(idleTimeout<=0关闭该行为),作为客户端治理失效时的最后防线。二、旧方案存在的问题(分·问题)
⭐ 1. 会把请求发到"已断开 / 正在关闭"的链接,导致报错(最关键)
原方案
ensureChannelActive在取 channel 前有重建检查,稳态下不会一直打死链;但在以下三类竞态/缺陷下会把请求打到断链上:NettyClientHandler收到 idle 事件直接channel.close()(异步派发到 EventLoop)。从"决定关闭"到"channel.isConnected()翻转为 false"之间有时间窗;窗口内请求线程isAvailable()仍返回 true → 判定无需重建 → 把请求发到正在关闭的 channel。channels用ArrayList,存在内存可见性 data race:getChannel0无锁读、ensureChannelActive锁内写,缺乏 happens-before,读线程可能读到过期 item,发到已被替换/已断开的旧 channel。send时可能已被 RST/FIN 断开;原方案无 TCP keepalive 快速探测、断连感知延迟大、窗口更宽(典型connection reset/IOException)。服务端主动 idle 关闭(发 FIN)同样是断链来源之一,但旧方案缺乏客户端侧的 invalidate-before-close 与快速探测来收窄这个窗口。2. 空闲即关闭,长连接形同虚设
服务端/客户端 Handler 的
userEventTriggered直接 close;RpcClusterClientManager还按lastUsedNanos关"长时间未使用"的客户端——对 Netty 客户端尤其致命(牵连共享EventLoopGroup、打断在途请求)。问题不在于"服务端是否该有空闲兜底",而在于旧方案空闲阈值过激、无法关闭、且客户端侧毫无配合(不失效 slot、不快速探测),使正常长连接被无谓拆除。3. 半死连接(half-dead)无法探测
"持续写 + 静默丢包"场景,客户端一直能写进内核缓冲区,
ALL_IDLE/WRITE_IDLE永不触发,完全感知不到对端已死。4. 其它并发竞态
TIME_WAIT。DefClusterInvoker用非 CAS 的invokerCache.remove(key)→ 可能误删别的线程刚装好的新 proxy。5. epoll 与共享 IO 线程组互斥
旧代码只有一个共享
NioEventLoopGroup,开 epoll 就被迫关ioThreadGroupShare。6. HTTP 连接池几乎裸奔
maxConns没生效(退化到 25/5)、无validateAfterInactivity(stale/NoHttpResponseException)、无空闲驱逐(fd 泄漏)、服务端不返回Keep-Alive头时按无限保活被 NAT/LB 丢弃、黑洞场景无兜底。7. 默认值不合理
maxConnections=20480(过大)、connsPerAddr=2。三、当前解决方案(分·方案)
A. tRPC 协议(Netty 真长连接)
⭐ 1. 消除"发到断链"竞态(对应问题 1)
invalidateChannel把 slot 置空(下次必然重连),再异步ctx.close(),使请求线程立刻看到"需要重连",消除 (a) 窗口。该机制同样收窄"服务端主动 idle 关闭"这类对端关闭引发的窗口——一旦客户端 READ-idle 或channelInactive感知到断连,slot 会被置空/翻转为不可用,下次请求走懒重连。channels:ArrayList→CopyOnWriteArrayList,提供 volatile 可见性,修复 (b) race。2. 移除客户端粗暴关闭,Netty 连接常驻;服务端保留可配置空闲兜底
NettyClientHandler中 idle 直接 close 的逻辑,客户端不再因空闲主动关连接(改由 READ-idle 半死探测驱动,见 3)。NettyTcpServerTransport保留IdleStateHandler:当idleTimeout>0时安装ALL_IDLE(默认DEFAULT_SERVER_IDLE_TIMEOUT=240000ms),NettyServerHandler.userEventTriggered收到IdleStateEvent后关闭连接;idleTimeout<=0(含null)则不安装该 handler、服务端永不主动断链。 此为兜底策略:仅当收发均长时间静默才触发,正常有流量或客户端已先行治理时不会命中。RpcClusterClientManager对 Netty 客户端不再按空闲关闭,随BackendConfig生命周期常驻。3. 客户端 READ idle 关闭 + 懒重连
NettyTcpClientTransport装 READ idle 的IdleStateHandler(默认 180000ms):只有"长时间收不到回包"才触发,正对应半死连接。[long-link][idle-fire]/[idle-close]运维日志。4. TCP keepalive 调优 + epoll 解耦
tcpKeepAliveIdle/Intvl/Cnt(30/10/3),~60s 内被内核 RST。5. 惊群防护
ensureChannelActive:connLock内双重检查 +needsReconnect状态机(notYetConnect/connecting/available),杜绝重连风暴。B. 集群侧(统一缓存治理 + 非 Netty 兜底)
idleTimeout无成功响应才关;用 in-flight 计数 + 单次 CAS 抢占 + CAS 后复查时间戳 三重手段把竞态窗口收到最窄,避免误杀在途请求。remove(key, value)),closeFuture 回调摘除后下次请求懒重建,避免误删新 proxy。C. HTTP / HTTP2 协议(连接池保活,无私有健康信号)
isAvailable()回退为父类AbstractRpcClient.isAvailable()(即lifecycleObj.isStarted()):客户端被集群 idle-scanner 关闭后 lifecycle 翻转,isAvailable()自然返回 false,DefClusterInvoker下次请求懒重建。evictIdle(按idleTimeout主动驱逐空闲连接)+ 集群 30s 扫描器(同样按idleTimeout关闭长时间无成功响应的整客户端),两层都使用同一份idleTimeout配置,语义统一、可配置。validateAfterInactivity(复用前校验)+evictExpired+SO_KEEPALIVE(OS 兜底)。⭐ HTTP 连接池配置(新增小节)
三个客户端(
HttpRpcClient=HTTP/1.1 走 HttpClient 4.x;Http2cRpcClient=h2c、Http2RpcClient=h2/TLS,均走 HttpClient 5.x async)统一的连接池参数:maxConnTotal/maxConnPerRouteprotocolConfig.getMaxConns()validateAfterInactivityHttpConstants.VALIDATE_AFTER_INACTIVITY_MS= 5000msNoHttpResponseException)evictExpiredConnectionsevictIdleConnectionsprotocolConfig.getIdleTimeout()(毫秒,默认 180000=180s);null或<=0则不启用keepAliveStrategyKeep-Alive: timeout=N封顶 5min,缺省也兜底 5minSO_KEEPALIVEtrueconnectionTimeToLiveD. 默认值与配置
maxConnections 20480 → 200、connsPerAddr 2 → 4(注意服务端入向连接翻倍)、新增 keepalive 默认值;BaseProtocolConfig与 SpringAbstractProtocolSchema打通tcp_keep_alive_*配置。idleTimeout默认DEFAULT_IDLE_TIMEOUT="180000"(ms,READ-idle 与 HTTP 池 evictIdle 共用);服务端idleTimeout默认DEFAULT_SERVER_IDLE_TIMEOUT="240000"(ms,ServiceConfig专属,透传到服务端ProtocolConfig驱动ALL_IDLE兜底)。HttpConstants.VALIDATE_AFTER_INACTIVITY_MS=5000;空闲驱逐复用BaseProtocolConfig.idleTimeout。四、总结(总)
主线一句话:从"pull 模型 + 全局粗暴空闲关闭"升级为"真长连接 + 分协议分层治理 + 懒重连 + 竞态收窄",并保留服务端一层可配置、可关闭的空闲兜底。
CopyOnWriteArrayList、READ-idle+keepalive;对端关闭/服务端兜底关闭亦经channelInactive+懒重连自愈ALL_IDLE兜底(默认 240s,idleTimeout<=0关闭);HTTP 池 evict + 集群扫描兜底ALL_IDLE兜底validate(5s)、evictExpired+evictIdle(idleTimeout)、keepalive 封顶(HTTP/1.1)、SO_KEEPALIVE;无私有健康信号、无硬 TTL最终效果:正常流量下连接稳定复用不再被无谓拆除;"正在关闭/对端已关"的竞态窗口里不再误发请求;异常(半死/RST/黑洞)能在有界时间内被探测并自愈;服务端保留一层可配置的空闲兜底(默认 240s,可关)以覆盖客户端治理失效的极端场景;HTTP 空闲治理由连接池与集群扫描按统一
idleTimeout协同。配套约 4000+ 行测试覆盖上述竞态与恢复场景。五、核心代码块清单
⭐ 消除"发到断链"竞态
1. invalidate-before-close(客户端 idle:先失效 slot 再异步关)
2.
invalidateChannel:锁内复读置空 slot(AbstractClientTransport)3.
channels改CopyOnWriteArrayList(修复可见性 race)4. 懒重连 + 惊群防护(锁内双检 + 状态机)
tRPC 长连接其它核心
5. READ-idle handler 安装 + TCP keepalive 调优
NettyTcpClientTransport(idleTimeout 默认 180000ms;keepalive 30/10/3)。6. 服务端可配置
ALL_IDLE空闲兜底NettyTcpServerTransport.initChannel:idleTimeout>0时安装server-idle(ALL_IDLE,默认 240000ms),<=0(含 null)不安装;resolveIdleTimeoutMills()归一 null/≤0 为 0 并防自动拆箱 NPE;NettyServerHandler.userEventTriggered收到IdleStateEvent后channel.close()。7. NIO/EPOLL 双共享组(引用计数幂等)
NettyAbstractClientTransport。集群侧治理
8. 非 Netty 空闲关闭:三重防竞态
9. in-flight 计数 + 仅成功响应刷新活跃时间
RpcClusterClientManager.ConsumerInvokerProxy.invoke。10. CAS 摘除缓存(防误删新 proxy)
DefClusterInvoker/RpcClusterClientManager(invokerCache.remove(key, value)+ closeFuture 回调)。HTTP / HTTP2
11. HTTP/1.1 连接池治理(HttpClient 4.x,无 TTL、evict 由 idleTimeout 驱动)
12. h2c / h2(TLS) 连接池治理(HttpClient 5.x async,共享
applyIdleEviction)13. HTTP 健康信号机制:已删除
原
HttpRpcClient.isAvailable()(连续失败 ≥50 / 空闲 >10min 报不可用)及markUsed/markSuccess/markFailure、consecutiveFailures/lastUsedNanos字段、HttpConsumerInvoker/Http2ConsumerInvoker中的相应调用全部移除。HTTP 客户端的可用性判断回归lifecycleObj.isStarted(),空闲下线由集群 30s 扫描器统一负责。默认值
14.
Constants/HttpConstants关键默认值以上是根据当前分支修改后的完整文档。主要更新点集中在 服务端空闲兜底这一处与代码的偏差:
ALL_IDLE兜底(idleTimeout>0安装,默认 240s;<=0关闭)",并新增两端 idleTimeout 配比建议。DEFAULT_SERVER_IDLE_TIMEOUT="240000",并区分客户端(180s)/服务端(240s)语义。NettyTcpServerTransport.initChannel+NettyServerHandler.userEventTriggered实际代码。其余(客户端 READ-idle、
invalidate-before-close、CopyOnWriteArrayList、惊群双检、集群三重防竞态、HTTP 池 evict/无 TTL/无私有健康信号、其余默认值)经核对与当前分支一致,保持不动。