#370's _gopy_clear_go_tls (movq $0, %fs:-8) hard-crashes CPython 3.12+ in a shipping extension

Thanks for gopy. This is a follow-up to #370 and its fix, PR #393 ("Fix memory leaks & crashes when loading multiple Go extensions in one Python process", merged 2026-06-02), which introduced _gopy_clear_go_tls(). That workaround is now a hard, deterministic crash for a real, released project. slidge-whatsapp (a WhatsApp/XMPP gateway, https://codeberg.org/slidge/slidge-whatsapp) segfaults CPython on the first call into its gopy extension. Full downstream investigation: https://codeberg.org/slidge/slidge-whatsapp/issues/153

PR #393 solved a real multi-extension case, but the TLS clear is applied unconditionally, and in the single-extension case (which is the common one) it does the opposite: it corrupts CPython's own thread state. Concrete details below; I am glad to test any patch.

Reproduction (real project, deterministic, no network/credentials)

python3.12 -m venv venv
venv/bin/pip install slidge-whatsapp==0.4.0b1      # ships a gopy-built wheel
venv/bin/python -c "import slidge_whatsapp.generated.whatsapp as w; w.NewGateway()"

Segmentation fault (core dumped)      # exit 139, every time, on the very first call

Reproduced on Gentoo, glibc, distro CPython 3.12 / 3.13 / 3.14 (identical), go 1.26.x, with gopy pinned at v0.4.11-0.20241206185020-5f285b890023. It does NOT crash on slidge-whatsapp 0.3.11b1, which predates _gopy_clear_go_tls (zero occurrences of it, of RequestGC, _gcReq, gc.callbacks); every build after it crashes.

Backtrace

Thread 1 "python" received signal SIGSEGV
#0  _PyType_GetDict               (libpython3.12.so.1.0)
#1  _PyObject_GenericGetAttrWithDict
#3  PyObject_GetAttr
#4  _PyEval_EvalFrameDefault
...
si_addr = 0x10
rip     = _PyType_GetDict+48:  mov 0x10(%rdx),%rcx
rdx     = 0x0                   # NULL

rdx is the current thread-state / interpreter pointer. It is NULL right after the CGo call returns and Python bytecode resumes, so 0x10(%rdx) faults at address 0x10 (the "segfault at 10" seen in dmesg).

Root cause

The generated Python wrapper calls _whatsapp._gopy_clear_go_tls() before every CGo entry. On linux/amd64 its body is:

static void _gopy_clear_go_tls(void) {
#elif defined(__x86_64__) && defined(__linux__)
    __asm__ volatile("movq $0, %%fs:-8" ::: "memory");
#endif
}

CPython 3.12 moved the current-thread-state fast path into a thread-local variable (_Py_tss_tstate, a _Py_thread_local PyThreadState *). It lives in the static TLS block, i.e. at a negative offset from the %fs thread pointer that the linker/loader assigns. The comment on _gopy_clear_go_tls assumes %fs:-8 is Go's private "current g" slot; that is true for Go's internal TLS layout, but when Go is built c-shared and loaded through glibc, the g pointer is reached via a linker-assigned TLS variable (runtime.tls_g), not a fixed -8. glibc owns the TCB and the static TLS block; Go's g slot and CPython's _Py_tss_tstate both live there at loader-chosen offsets. Writing a hardcoded %fs:-8 therefore clobbers whatever thread-local sits at that offset: here, CPython's current thread state, so _PyThreadState_GET() returns NULL and the interpreter crashes.

Why it slips through CI ("works on my machine")

The colliding offset depends on the exact static-TLS layout, so the same .so crashes on one CPython build and not another. slidge-whatsapp's CI builds the wheel against a standalone CPython (the cgo preamble hardcodes -I/root/.local/share/uv/python/...), which is also CI's runtime, and there %fs:-8 does not collide. Users install that same wheel against their distro CPython, where it does. That is exactly why #153 kept getting "cannot reproduce" while some users on a distro Python hit it.

Proof it is this instruction

Overwriting the single movq $0, %fs:-8 (bytes 64 48 c7 04 25 f8 ff ff ff 00 00 00 00, 13 bytes) with NOPs in the exact crashing .so removes the crash entirely; the extension then runs under real traffic. I fixed it for my particular installation using a post-build patch, but I believe it is a workaround for something that should be handled in gopy.

Possible directions (suggestions)

Make the TLS clear opt-in, or skip it unless more than one gopy runtime is actually present, so the single-extension case (probably the majority) is safe by default.
Drop the hardcoded %fs:-8 / %gs:0x30. In c-shared mode the g pointer is at a linker-assigned offset (runtime.tls_g), so -8 is not guaranteed to be Go's slot.
If the intent is to force needm() for a foreign thread, prefer a mechanism that does not poke raw TLS from C (e.g. pin with runtime.LockOSThread on the entry goroutine, or let needm() establish context per entry).

Happy to build test binaries or run patches against the slidge-whatsapp repro above.

@satarsa

Hi @satarsa and I'm sorry for the slow reply. This is a "nights and weekends" effort for me (and others), so I will do my best. I'm optimistic that we will be able to solve this working together.

I also want to acknowledge and sincerely thank you for this detailed bug report, it is a tremendous help. 🙇

I'm getting 503 errors for Codeberg at the moment, so I'm not able to see the full context you're linking to, but I'll try again over the coming days.

I really appreciate your list of possible solutions. Since some time has gone by, I'm wondering if you have a preferred solution. Additionally, I wonder if you might have a test case that you would be able to provide or something in a Pull Request that we could use to have more granular discussion.

@satarsa

@satarsa I'm sorry for the spam, I just wanted to mention one more thing hoping that it might be helpful. Specifically, I was thinking that you might want to take a look at #394 , just as a reference example of a narrowly scoped issue & fix.

Tracing the output of the fix (for PR 394) is a little tricky. You won't find the literal print() text like "Go empty bytes to Python: b'' anywhere in a green CI log, you'd only see that in a failing test. That's because of how testPkgBackend in main_test.go works (line ~1114): it runs the script with cmd.CombinedOutput(), captures output into buf, and does reflect.DeepEqual(got, want). If they match, buf is discarded — nothing is ever written to t.Log or stdout. The only thing that unconditionally prints is the generic fmt.Printf("running %s test.py\n", pyvm).

@b-long

Thanks @b-long, and no worries about the pace at all: 0% judgement, 100% understanding. Answering your two questions directly: my preferred solution is option 1 (make the clear opt-in, off by default), and below is a small patch plus the tests/verification that back it. Happy to turn this into a PR whenever you like.

My preferred solution: make `_gopy_clear_go_tls` opt-in, default off

There are two independent problems with the current unconditional clear:

It runs even when a single gopy extension is loaded, which is the common case and the one that cannot have a cross-runtime collision. There it is pure downside.
The address it writes is wrong for c-shared under glibc. In a normal Go binary the runtime lays out its TLS so the g pointer sits at %fs:-8, but in a c-shared library loaded by glibc the loader owns the static TLS block and the g pointer is reached through a linker-assigned TLS variable (runtime.tls_g), not a fixed -8. So movq $0, %fs:-8 does not clear Go's g at all; it zeroes whatever thread-local happens to sit there. On CPython 3.12+ that is _Py_tss_tstate, hence the crash.

The key point is that PR #393 already shipped the mechanism that actually isolates the runtimes: it loads each extension without RTLD_GLOBAL, and gopy links each .so with a version script that exports only PyInit_*. Both of these keep every Go runtime symbol (including the g-pointer TLS) local to its own .so, so nothing is shared to collide over. The raw-TLS clear is therefore redundant, and with the wrong offset it is actively harmful. Making it opt-in keeps the belt-and-suspenders available for anyone who wants it, while making the default safe.

Suggested patch (against current master, `72557f6`)

Adds a -clear-go-tls build flag (default false) and gates the single call site on it. The C helper and its pybindgen registration are left in place so opt-in restores today's behavior exactly; only the emitted call before each CGo entry is now conditional. The RTLD_GLOBAL handling is untouched.

diff --git a/bind/gen.go b/bind/gen.go
index 97c1ade..f6d8604 100644
--- a/bind/gen.go
+++ b/bind/gen.go
@@ -329,12 +329,13 @@ cwd = os.getcwd()
 currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
 os.chdir(currentdir)
 # When multiple gopy extensions coexist in one Python process each carries its own
-# independent Go runtime. The Go extension is loaded without RTLD_GLOBAL below, and
-# _gopy_clear_go_tls() is called before each CGo entry to force needm() to run, which
-# establishes the correct per-extension M/P/mcache context (issue #370).
-# Also load the extension without RTLD_GLOBAL so that Go runtime symbols stay
-# local to each .so — belt-and-suspenders on platforms where RTLD_GLOBAL is the
-# Python default (e.g. some Linux builds).
+# independent Go runtime. Loading each extension without RTLD_GLOBAL below keeps its
+# Go runtime symbols (including the per-runtime goroutine-pointer TLS slot) local to
+# its own .so, which is what prevents the runtimes from colliding (issue #370).
+# A _gopy_clear_go_tls() call before each CGo entry is available as an extra safety
+# net but is opt-in (gopy build -clear-go-tls), off by default: on glibc + CPython
+# 3.12+ its hardcoded TLS store clobbers the interpreter thread state and crashes on
+# the first call in the common single-extension case (issue #395).
 if hasattr(sys, 'getdlopenflags'):
 	try:
 		import ctypes as _gopy_ctypes
@@ -518,6 +519,17 @@ var NoWarn = false
 // NoMake turns off generation of Makefiles
 var NoMake = false
 
+// ClearGoTLS controls whether generated wrappers emit a _gopy_clear_go_tls()
+// call before every CGo entry point (issue #370). It is opt-in and off by
+// default. The clear performs a hardcoded TLS store (movq $0, %gs:0x30 on
+// darwin/amd64, movq $0, %fs:-8 on linux/amd64); with a single gopy extension
+// in the process it is unnecessary, and on glibc + CPython 3.12+ that offset
+// overlaps the TLS slot CPython uses for the current thread state, so the store
+// nulls it and the interpreter segfaults on the first call (issue #395).
+// Loading each extension without RTLD_GLOBAL, done unconditionally, already
+// keeps each runtime's goroutine-pointer TLS local to its own .so.
+var ClearGoTLS = false
+
 // GenPyBind generates a .go file, build.py file to enable pybindgen to create python bindings,
 // and wrapper .py file(s) that are loaded as the interface to the package with shadow
 // python-side classes
diff --git a/bind/gen_func.go b/bind/gen_func.go
index fc2ee15..b264334 100644
--- a/bind/gen_func.go
+++ b/bind/gen_func.go
@@ -339,8 +339,13 @@ if __err != nil {
 	// Clear the Go TLS goroutine slot before the CGo entry point so that
 	// Go's needm() runs and establishes the correct per-extension context.
 	// Without this, two extensions sharing the same process can corrupt
-	// each other's heap via TLS collision (issue #370).
-	g.pywrap.Printf("_%s._gopy_clear_go_tls()\n", pkgname)
+	// each other's heap via TLS collision (issue #370). Opt-in and off by
+	// default: the hardcoded TLS store crashes CPython 3.12+ in the common
+	// single-extension case (issue #395), and RTLD_GLOBAL-local loading
+	// already isolates each runtime's goroutine-pointer TLS.
+	if ClearGoTLS {
+		g.pywrap.Printf("_%s._gopy_clear_go_tls()\n", pkgname)
+	}
 
 	// pywrap output
 	mnm := fsym.ID()

The same two lines (flag declaration + bind.ClearGoTLS = ...) are added to each of cmd_build.go, cmd_gen.go, cmd_pkg.go and cmd_exe.go, mirroring the existing no-make flag, e.g.:

@@ cmd_build.go
 	cmd.Flag.Bool("no-make", false, "do not generate a Makefile, e.g., when called from Makefile")
+	cmd.Flag.Bool("clear-go-tls", false, "emit a _gopy_clear_go_tls() call before every CGo entry (issue #370); off by default, needed only when several gopy extensions share one process and known to crash CPython 3.12+ (issue #395)")
 	cmd.Flag.Bool("dynamic-link", false, "whether to link output shared library dynamically to Python")
@@
 	bind.NoWarn = cfg.NoWarn
 	bind.NoMake = cfg.NoMake
+	bind.ClearGoTLS = cmdr.Flag.Lookup("clear-go-tls").Value.Get().(bool)

gofmt and go vet ./... are clean, and go test ./bind/ passes.

Verification on the real downstream project

Environment: Gentoo, glibc, distro CPython 3.14.6, go 1.26.4, gcc 16, gopy master 72557f6. Rebuilding slidge-whatsapp's extension with the patched gopy:

Default (clear off): the generated whatsapp.py contains zero _gopy_clear_go_tls() call sites, and the first call now succeeds:

NewGateway OK -> whatsapp.Gateway ( ... handle=1 )   # exit 0

gopy build -clear-go-tls (opt-in): the call reappears (30 sites, one per CGo entry) and the crash returns exactly as before:

Segmentation fault      # exit 139

So the flag is a faithful toggle of the old behavior, and off-by-default removes the crash with no post-build patching.

Why off-by-default is safe for the multi-extension case too

Two independent checks, both with the clear off:

Coexistence stress. Two independent gopy extensions (two separate Go runtimes) loaded in one interpreter, 200000 interleaved calls into both on the same OS thread with periodic gc.collect() (the exact condition "broken" since go 1.21 #370 describes):

both imported under python 3.14.6
iterations done, wrong results: 0
PASS: two Go runtimes coexisted, no crash, no corruption

Symbol isolation. Each .so exports exactly one global dynamic symbol and no Go runtime symbols at all:

_simple.....so : exported GLOBAL non-UND dynsyms = 1  -> PyInit__simple   (runtime.* exported: 0)
_simpleb....so : exported GLOBAL non-UND dynsyms = 1  -> PyInit__simpleb  (runtime.* exported: 0)

Because runtime.tls_g and friends are local to each .so, the g pointer is per-runtime-private and cannot be interposed across extensions, independently of RTLD_GLOBAL. This is what removes the premise behind the clear.

I have to emphasize: I do not have the original #370 multi-extension reproducer, so I cannot claim to have re-run that exact case. The opt-in keeps the old behavior one flag away for anyone who needs it, and I am glad to validate against the #370 repro if you can point me at it.

Suggested regression tests

Matching how testPkgBackend runs (python test.py, compare output, a segfault shows up as a non-zero exit):

Codegen assertion (deterministic, layout independent): generate a package with the default flags and assert the generated wrapper does not contain _gopy_clear_go_tls(), and with -clear-go-tls that it does. This guards the default without depending on any particular CPython TLS layout.
Coexistence example: a small _examples package pair built as two extensions, imported together, cross-called under gc.collect(), asserting a clean run. This is the multi-runtime path that Fix memory leaks & crashes when loading multiple Go extensions in one Python process #393 was protecting; keeping it green with the clear off documents that the isolation comes from RTLD_GLOBAL-local loading plus the version script, not from the TLS poke.

I can include both in the PR.

Downstream slidge-whatsapp

This morning (8 UTC) when I was reading your responses, codeberg was really down (such an irony...), now it seems to be back in operation. For slidge-whatsapp (https://codeberg.org/slidge/slidge-whatsapp/issues/153): until a gopy release carries this, the interim fix is a post-build NOP of the single movq $0, %fs:-8 (13 bytes -> NOPs), which is already running in production at my server. Once this lands, the clean path is to drop that patch and simply build with the default (clear off). And I can forget about it, finally and forever...

@satarsa

@satarsa Thank you for the detailed message. I think this is a reasonable approach, and if you can open a PR that'd be a massive help.

I have an idea about an independent reproducer (build and execute) test, and when I find time to experiment I'll be sure to share the details.

I'll watch for your PR, thank you again.

bind: make _gopy_clear_go_tls opt-in, off by default

feat: add '3.12' testing & improve complex conversion (#399)

Navigation Menu

Repository navigation

#370's _gopy_clear_go_tls (movq $0, %fs:-8) hard-crashes CPython 3.12+ in a shipping extension #395

Reproduction (real project, deterministic, no network/credentials)

Backtrace

Root cause

Why it slips through CI ("works on my machine")

Proof it is this instruction

Possible directions (suggestions)

My preferred solution: make `_gopy_clear_go_tls` opt-in, default off

Suggested patch (against current master, `72557f6`)

Verification on the real downstream project

Why off-by-default is safe for the multi-extension case too

Suggested regression tests

Downstream slidge-whatsapp

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

#370's _gopy_clear_go_tls (movq $0, %fs:-8) hard-crashes CPython 3.12+ in a shipping extension #395

Description

Reproduction (real project, deterministic, no network/credentials)

Backtrace

Root cause

Why it slips through CI ("works on my machine")

Proof it is this instruction

Possible directions (suggestions)

Activity

b-long commented on Jul 22, 2026

b-long commented on Jul 22, 2026

satarsa commented on Jul 22, 2026

My preferred solution: make _gopy_clear_go_tls opt-in, default off

Suggested patch (against current master, 72557f6)

Verification on the real downstream project

Why off-by-default is safe for the multi-extension case too

Suggested regression tests

Downstream slidge-whatsapp

b-long commented on Jul 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

My preferred solution: make `_gopy_clear_go_tls` opt-in, default off

Suggested patch (against current master, `72557f6`)