Sentry: store_raft.go:557: log.Dev.Fatal: while applying snapshot: while ingesting [× × × × × ×] and excising /System/NodeLivenessMax-/System/tsd: link × ×: operation not permitted (1) attache...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/7464290188/?referrer=webhooks_plugin

Panic Message:

store_raft.go:557: log.Dev.Fatal: while applying snapshot: while ingesting [× × × × × ×] and excising /System/NodeLivenessMax-/System/tsd: link × ×: operation not permitted
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
  | 	pkg/kv/kvserver/store_raft.go:557
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
  | 	pkg/kv/kvserver/store_raft.go:433
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
  | 	pkg/kv/kvserver/store_raft.go:497
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
  | 	pkg/kv/kvserver/store_snapshot.go:602
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
  | 	pkg/kv/kvserver/store_raft.go:283
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	pkg/util/stop/stopper.go:351
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
  | 	pkg/kv/kvserver/store_raft.go:280
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).raftSnapshot
  | 	pkg/kv/kvserver/raft_transport.go:567
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
  | 	pkg/kv/kvserver/raft_transport.go:545
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
  | 	bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
  | 	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:123
  | google.golang.org/grpc.getChainStreamHandler.func1
  | 	external/org_golang_google_grpc/server.go:1504
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
  | 	pkg/rpc/context.go:190
  | google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1
  | 	external/org_golang_google_grpc/server.go:1504
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
  | 	pkg/rpc/auth.go:193
  | google.golang.org/grpc.getChainStreamHandler.func1
  | 	external/org_golang_google_grpc/server.go:1504
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
  | 	pkg/rpc/context.go:147
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	pkg/util/stop/stopper.go:351
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
  | 	pkg/rpc/context.go:146
  | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
  | 	external/org_golang_google_grpc/server.go:1495
  | google.golang.org/grpc.(*Server).processStreamingRPC
  | 	external/org_golang_google_grpc/server.go:1659
  | google.golang.org/grpc.(*Server).handleStream
  | 	external/org_golang_google_grpc/server.go:1739
  | google.golang.org/grpc.(*Server).serveStreams.func1.1
  | 	external/org_golang_google_grpc/server.go:970
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1693
Wraps: (2) secondary error attachment
  | while applying snapshot: while ingesting [× × × × × ×] and excising /System/NodeLivenessMax-/System/tsd: link × ×: operation not permitted
  | (1) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |   | 	pkg/kv/kvserver/replica_raft.go:1161
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
  |   | 	pkg/kv/kvserver/store_raft.go:556
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
  |   | 	pkg/kv/kvserver/store_raft.go:433
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
  |   | 	pkg/kv/kvserver/store_raft.go:497
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
  |   | 	pkg/kv/kvserver/store_snapshot.go:602
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
  |   | 	pkg/kv/kvserver/store_raft.go:283
  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  |   | 	pkg/util/stop/stopper.go:351
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
  |   | 	pkg/kv/kvserver/store_raft.go:280
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).raftSnapshot
  |   | 	pkg/kv/kvserver/raft_transport.go:567
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
  |   | 	pkg/kv/kvserver/raft_transport.go:545
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
  |   | 	bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13
  |   | 	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:123
  |   | google.golang.org/grpc.getChainStreamHandler.func1
  |   | 	external/org_golang_google_grpc/server.go:1504
  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
  |   | 	pkg/rpc/context.go:190
  |   | google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1
  |   | 	external/org_golang_google_grpc/server.go:1504
  |   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
  |   | 	pkg/rpc/auth.go:193
  |   | google.golang.org/grpc.getChainStreamHandler.func1
  |   | 	external/org_golang_google_grpc/server.go:1504
  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
  |   | 	pkg/rpc/context.go:147
  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  |   | 	pkg/util/stop/stopper.go:351
  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
  |   | 	pkg/rpc/context.go:146
  |   | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2
  |   | 	external/org_golang_google_grpc/server.go:1495
  |   | google.golang.org/grpc.(*Server).processStreamingRPC
  |   | 	external/org_golang_google_grpc/server.go:1659
  |   | google.golang.org/grpc.(*Server).handleStream
  |   | 	external/org_golang_google_grpc/server.go:1739
  |   | google.golang.org/grpc.(*Server).serveStreams.func1.1
  |   | 	external/org_golang_google_grpc/server.go:970
  | Wraps: (2) while applying snapshot
  | Wraps: (3) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshotRaftMuLocked
  |   | 	pkg/kv/kvserver/replica_raftstorage.go:670
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |   | 	pkg/kv/kvserver/replica_raft.go:1160
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
  |   | 	pkg/kv/kvserver/store_raft.go:556
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
  |   | 	pkg/kv/kvserver/store_raft.go:433
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
  |   | 	pkg/kv/kvserver/store_raft.go:497
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
  |   | 	pkg/kv/kvserver/store_snapshot.go:602
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
  |   | 	pkg/kv/kvserver/store_raft.go:283
  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  |   | 	pkg/util/stop/stopper.go:351
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
  |   | 	pkg/kv/kvserver/store_raft.go:280
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).raftSnapshot
  |   | 	pkg/kv/kvserver/raft_transport.go:567
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
  |   | 	pkg/kv/kvserver/raft_transport.go:545
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
  |   | 	bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267
  |   | github.com/c...

Stacktrace (expand for inline code snippets):

src/runtime/asm_amd64.s#L1692-L1694
external/org_golang_google_grpc/server.go#L969-L971
external/org_golang_google_grpc/server.go#L1738-L1740
external/org_golang_google_grpc/server.go#L1658-L1660
external/org_golang_google_grpc/server.go#L1494-L1496
pkg/rpc/context.go#L145-L147
pkg/util/stop/stopper.go#L350-L352
pkg/rpc/context.go#L146-L148
external/org_golang_google_grpc/server.go#L1503-L1505
pkg/rpc/auth.go#L192-L194
external/org_golang_google_grpc/server.go#L1503-L1505
pkg/rpc/context.go#L189-L191
external/org_golang_google_grpc/server.go#L1503-L1505
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go#L122-L124
bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/https://github.com/cockroachdb/cockroach/blob/ea8d70f63642a8cd69002a2baedc47fa077239cb/pkg/kv/kvserver/storage_services.pb.go#L266-L268
pkg/kv/kvserver/raft_transport.go#L544-L546
pkg/kv/kvserver/raft_transport.go#L566-L568
pkg/kv/kvserver/store_raft.go#L279-L281
pkg/util/stop/stopper.go#L350-L352
pkg/kv/kvserver/store_raft.go#L282-L284
pkg/kv/kvserver/store_snapshot.go#L601-L603
pkg/kv/kvserver/store_raft.go#L496-L498
pkg/kv/kvserver/store_raft.go#L432-L434
pkg/kv/kvserver/store_raft.go#L556-L558

src/runtime/asm_amd64.s in runtime.goexit at line 1693
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.1 at line 970
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1739
external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1659
external/org_golang_google_grpc/server.go in google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 at line 1495
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 146
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 351
pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 147
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/rpc/auth.go in pkg/rpc.kvAuth.streamInterceptor at line 193
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1 at line 1504
pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 190
external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1504
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/rpc.NewServerEx.StreamServerInterceptor.func13 at line 123
bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go in pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler at line 267
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).RaftSnapshot at line 545
pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).raftSnapshot at line 567
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot at line 280
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 351
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot.func1 at line 283
pkg/kv/kvserver/store_snapshot.go in pkg/kv/kvserver.(*Store).receiveSnapshot at line 602
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest at line 497
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).withReplicaForRequest at line 433
pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1 at line 557

Tags

Tag	Value
Command	server
Environment	v26.1.3
Go Version	go1.25.5
Platform	linux amd64
Distribution	CCL
Cockroach Release	v26.1.3
Cockroach SHA	ea8d70f
# of CPUs	16
# of Goroutines	722

Jira issue: CRDB-63688

[blathers-crl]

CC'ing via the CODEOWNERS-based sentry heuristic:

• @cockroachdb/kv

Sentry issue cause: pkg/kv/kvserver/store_raft.go

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}