perf: move EG() and CG() in ZTS builds into __thread storage by henderkes · Pull Request #23227 · php/php-src

henderkes · 2026-08-12T12:12:02Z

replay of #22231

Moves EG and CG into __thread storage after all. We first moved them into constant offsets (#22287) from *_tsrm_ls_cache, but I couldn't find a way to stop gcc or clang from reloading _tsrm_ls_cache base pointer between function calls, leading to an extra pointer load once per function.

This eliminates the pointer load, making the access sequence to EG/CG just a single mov (x64) / mrs + add + ldr (aarch64) under local-exec. initial-exec likewise loses the pointer load so 3 -> 2 instructions (x64).
cc @arnaud-lb

What I'm adding here to counter the global-dynamic fallback slowdown is the option to explicitly opt-in to initial-exec, under the knowledge that host programs loading it will need to increase the static tls surplus. This is not an issue for package providers.

PS: Actually figured out that the explicit model choice doesn't happen for musl, not sure about IE (static tls surplus on musl?), but LE should work just fine. That's for another PR though.

…truct

… packagers want to set the glibc tls surplus in their httpd package

…s are in TLS storage)

arnaud-lb

I like the --with-tsrm-tls-model idea.

Did a first pass, but I will take the time to review carefully.

henderkes

It's a bit of a pain to find these on mobile, so I'll pin just these two, but there's more.

It could eventually be reworked when all the symbols move directly into thread storage, but I've not even begun thinking about it. There's probably no point except for zend_ini_scanner_globals and what we still have in the front.

henderkes · 2026-08-14T14:55:01Z

+#if defined(ZEND_WIN32) && !defined(LIBZEND_EXPORTS)
+/* Windows can't dllexport __declspec(thread) symbols, so outside Zend each module
+ * keeps a per-module `void *` pointer and reaches EG/CG via the resource-id indirection. */
+# define ZEND_TSRMLS_CACHE_T void *


The symbol is referenced as just a void* on windows

henderkes · 2026-08-20T16:33:13Z

Created the aarch64 global-dynamic first first, but I may as well look into teaching the JIT to take the address of _tsrm_ls_cache + offset instead of the mandatory load for a bit of JIT speedup.

henderkes · 2026-08-20T16:35:50Z

@arnaud-lb out of

zend_ini_scanner_globals))
virtual_cwd_globals))
zend_signal_globals_t))
zend_gc_globals_size())
php_core_globals))
sapi_globals_struct))
zend_accel_globals))
zend_jit_globals))

Which do you think would make sense moving too? gc globals probably and maybe virtual_cwd_globals?

arnaud-lb · 2026-08-20T16:45:42Z

alloc_globals is likely the most accessed global as every emalloc/efree and related fetch AG(mm_heap). I know that zend_alloc_globals is not exposed, but we could reserve enough space in _tsrm_ls_cache without exposing it:

struct _zend_tsrm_ls_cache {
	void *cache;
	void *self;
	zend_executor_globals eg;
	zend_compiler_globals cg;
	uint8_t ag[123];
	...
};

// zend_alloc.c:
ZEND_STATIC_ASSERT(sizeof(zend_tsrm_ls_cache.ag) >= sizeof(zend_alloc_globals));

#define AG(v) (((zend_alloc_globals*)&_tsrm_ls_cache.ag)->v)

henderkes · 2026-08-25T10:06:59Z

Created the aarch64 global-dynamic first first, but I may as well look into teaching the JIT to take the address of _tsrm_ls_cache + offset instead of the mandatory load for a bit of JIT speedup.

Proving too far outside my expertise. I think this is reviewable like this now, the JIT optimization could be done later in IR side to let us drop the self reference.

Unless @dstogov would like to have a go at it.

arnaud-lb

The changes look good in general, but it's a bit hard to follow. Possibly grouping abstractions in TSRM would help? Right now you need to have TSRM.*, zend.c, zend_globals_macros.h to understand what's going on.

The Windows changes make sense, but maybe @shivammathur can take a look as well?

ts_free_resources(), ts_free_thread(), ts_free_id() and ts_apply_for_id() matched entries with p->thread_id == tsrm_thread_id(). That is ambiguous in exactly the case the surrounding code exists to handle: a stale entry of a dead thread can carry the live thread's recycled id. Compare against the entry that tsrm_tls_get() hands out instead. The recycle path in ts_resource_ex() no longer runs the stale entry's destructors. It used to point the TLS cache at an entry whose native TLS block had died with its thread, so any destructor reaching for EG or CG read freed memory. Leaking the dead thread's module globals is the better trade; a child process that recycles thread ids gets respawned by the SAPI anyway. Hold the shutdown marker across the window so that signal handlers stay away from both the stale entry and its replacement. Also reset tls_key, the TSRM tables and TSRMLS_CACHE on shutdown.

…t budget

henderkes · 2026-09-04T13:39:15Z

to me central position seems better, although I can see the appeal of trying to keep it inside the module

From @iliaal. I'll try approaching this with global per-thread cleanup hooks, but leave persistent_list in EG and wire ts_free_thread to clean it up.

…free_thread` at thread exit SAPI's should already been doing this on their own before, but the apache2handler doesn't get control over threads, so it didn't call ts_free_thread and therefore started leaking resources with the move of per-thread globals from heap storage to native __thread storage

henderkes · 2026-09-05T11:51:28Z

-	}
+	do {
+		zend_try {
+			zend_destroy_rsrc_list(&EG(persistent_list));


this is technically a bugfix, plist previously leaked on threads that unregistered with ts_free_thread().

henderkes · 2026-09-12T06:47:42Z

@iliaal could you take a look to see if the change looks acceptable to you?

iliaal · 2026-09-13T19:50:25Z

+		free(data);
+		return;
+	}
+	if (FlsGetValue(tsrm_exit_key) != data) {


Does FlsGetValue(tsrm_exit_key) still return data when this callback fires at thread exit? MSDN documents lpFlsData only as "the value stored in the FLS slot" and says nothing about whether the slot is cleared first. If it is cleared, this returns before ts_free_thread() and the Windows half of the automatic cleanup never runs. I have no Windows box to settle it.

Yes it runs, I've tested on musl, glibc and Windows.

Edit: clarifying wording, I've fully tested everything on musl, glibc and windows. Fls callbacks obviously only on Windows.

iliaal · 2026-09-13T19:50:36Z

+ZEND_API void zend_win_tsrm_cache_shutdown(void);
+ZEND_API zend_tsrm_ls_cache *zend_win_tsrm_cache_fallback(void);
+#  ifdef __clang__
+static __inline__ __attribute__((const, always_inline)) zend_tsrm_ls_cache *zend_win_tsrm_cache_ptr(void)


const is formally wrong here: the function reads zend_win_tsrm_cache_offset and gs: memory. It is harmless in practice since the published value is thread-invariant, and the CSE it enables is safer than re-reading gs after a TlsFree. __readgsqword under const gives the same instruction on clang-cl, so the clang branch and its asm could go and leave one implementation.

Formally wrong, but safe in our circumstances and it gets clang to cse where it otherwise wouldn't.

Technically we can always &_tsrm_ls_cache, it's just not optimal codegen.

__readgsqword under const gives the same instruction on clang-cl, so the clang branch and its asm could go and leave one implementation.

Not on PC to test this atm but I'm 98% sure it resulted in unnecessary reloads under Clang 21 and 22.

iliaal · 2026-09-13T19:50:37Z

+AS_VAR_IF([PHP_THREAD_SAFETY], [yes], [
+  AS_CASE([$PHP_TSRM_TLS_MODEL],
+    [auto], [
+      AS_VAR_IF([php_tsrm_dlopened_sapi], [yes],


apache2handler is the only SAPI setting this. sapi/embed also builds a libphp.so under --enable-embed=shared, so a host that dlopens it rather than linking against it gets initial-exec and the static TLS surplus failure this flag exists to avoid. Setting it there too, or stating that embed is link-time only, would close the gap.

I'm not aware of any embed users that don't link against it. If there are, I'm happy to declare embed sapi as dlopened too, I can always overwrite the model.

iliaal · 2026-09-13T19:50:38Z

- AG and SCNG are now allocated with ts_allocate_tls_id() and live in native
-  __thread storage on ZTS builds.
+- tsrm_shutdown() and the bundled library-unload hooks automatically disarm
+  thread-exit cleanup. Embedders must stop and join all PHP worker threads


Two cases this does not cover. A detached thread runs ts_free_thread() from the new exit callback but can never be joined, so "stop and join all PHP worker threads" is unsatisfiable for it. And when a thread is still alive here, what actually survives is its persistent PDO/mysqli/pgsql connections, which is worth naming since it is the visible consequence.

Poor wording, I think. Embedders should explicitly call ts_free_thread, only if they don't, the callbacks do the cleanup automatically, unless they call zend_shutdown before (which is an obviously terrible idea, but is unchanged by this PR).

iliaal · 2026-09-13T19:52:55Z

Nothing blocking from me, the inline notes are all nits.

henderkes added 7 commits August 12, 2026 12:57

perf: embed executor and compiler globals in the __thread TLS cache s…

b2dc217

…truct

fix typo

2a7884d

leave the option of initial-exec tls model even for apache2handler if…

2e248cb

… packagers want to set the glibc tls surplus in their httpd package

remove ts_allocate_fast_id_at again (unused, now that the four global…

4aef0ba

…s are in TLS storage)

guard ts_apply_for_id and ts_free_id for own threads globals too

4633dc0

reword comment

d67443a

simplify linux aarch64 jit generator

c509919

henderkes requested review from bukka and dstogov as code owners August 12, 2026 12:12

github-actions Bot added Category: Build System Category: Engine Extension: opcache SAPI: apache2handler ABI break labels Aug 12, 2026

stupid typo

bd1730d

arnaud-lb reviewed Aug 12, 2026

View reviewed changes

Comment thread ext/opcache/jit/zend_jit_ir.c Outdated

Comment thread TSRM/TSRM.h Outdated

Comment thread Zend/zend.c Outdated

henderkes added 4 commits August 12, 2026 17:35

clarification

fdc77a4

abort, rather than assert

06f5bba

largely revert jit, just offsetof self* in the tsrm struct

209979d

LE guard for external embedders

5d9e202

henderkes force-pushed the perf/1-tls-eg-cg branch from b78839c to 5d9e202 Compare August 13, 2026 19:43

henderkes commented Aug 14, 2026

View reviewed changes

henderkes added 4 commits August 20, 2026 17:33

drop unused TSRMLS_MAIN_CACHE_DEFINE()

6c5769e

spelling mistakes

b274ea5

remove useless line

6dd5443

apply IR fix again

edbb00b

henderkes mentioned this pull request Aug 20, 2026

allow global-dynamic on glibc dstogov/ir#181

Merged

arnaud-lb reviewed Aug 26, 2026

View reviewed changes

Comment thread Zend/zend_globals_macros.h Outdated

Comment thread Zend/zend_globals_macros.h Outdated

Comment thread Zend/zend_globals_macros.h Outdated

Comment thread Zend/zend_globals_macros.h Outdated

Comment thread Zend/zend.c Outdated

shivammathur reviewed Aug 26, 2026

View reviewed changes

Comment thread win32/dllmain.c

Comment thread Zend/zend.c

henderkes added 11 commits August 28, 2026 14:50

feedback: teb slot needs _m_x64

b628c09

feedback: precompute teb slot offset

a159bea

feedback: move macros into tsrm.h

5b62a60

feedback: add zend_tsrmg_direct macro

493dc16

feedback: move zend_win_tsrm_cache_init to tsrm.c

b8fdbe3

feedback: tsrm initializes tls slot

85ae370

perf: non-volatile gs read on clang

eb65a2f

fix tests

92763a3

fix gdb script

d05af7b

fall back to &_tsrm_ls_cache in case something blew the early TEB slo…

4c08c6c

…t budget

henderkes force-pushed the perf/1-tls-eg-cg branch from f5c7bcc to 4c08c6c Compare August 28, 2026 16:33

shivammathur reviewed Aug 28, 2026

View reviewed changes

Comment thread TSRM/TSRM.c Outdated

suggestion: TlsFree in case of dll unload or late-TEB fallback

3ed3ddd

henderkes added 2 commits September 4, 2026 23:16

let ts_free_thread() clear EG(persistent_list) to prevent leaks

50a3932

henderkes commented Sep 5, 2026

View reviewed changes

henderkes requested review from arnaud-lb and shivammathur September 5, 2026 11:51

Merge branch 'master' into perf/1-tls-eg-cg

cb003fc

iliaal reviewed Sep 13, 2026

View reviewed changes

Uh oh!

Conversation

henderkes commented Aug 12, 2026

Uh oh!

arnaud-lb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

henderkes left a comment

Choose a reason for hiding this comment

Uh oh!

henderkes Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

henderkes commented Aug 20, 2026

Uh oh!

henderkes commented Aug 20, 2026

Uh oh!

arnaud-lb commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henderkes commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arnaud-lb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

henderkes commented Sep 4, 2026

Uh oh!

henderkes Sep 5, 2026

Choose a reason for hiding this comment

Uh oh!

henderkes commented Sep 12, 2026

Uh oh!

iliaal Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

henderkes Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iliaal Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

henderkes Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iliaal Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

henderkes Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

iliaal Sep 13, 2026

Choose a reason for hiding this comment

Uh oh!

henderkes Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iliaal commented Sep 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

arnaud-lb commented Aug 20, 2026 •

edited

Loading

henderkes commented Aug 25, 2026 •

edited

Loading

henderkes Sep 13, 2026 •

edited

Loading

henderkes Sep 13, 2026 •

edited

Loading

henderkes Sep 13, 2026 •

edited

Loading