gh-122931: Include multiarch tuple in abi3t filenames by stefanor · Pull Request #122917 · python/cpython

stefanor · 2024-08-11T21:32:12Z

This permits abi3t stable ABI extensions for multiple architectures to be co-installed into the same directory, without clashing with each other, the same way (non-stable ABI) regular extensions can.

Issue: Allow stable ABI extensions to include multiarch tuples in the filename #122931

…ple in the filename This permits stable ABI extensions for multiple architectures to be co-installed into the same directory, without clashing with each other, the same way (non-stable ABI) regular extensions can. It is listed below the current .abi3 suffix because setuptools will select the first suffix containing .abi3, as the target filename. We do this to protect older Python versions predating this patch.

vstinner

You should document this change in https://docs.python.org/dev/whatsnew/3.14.html (Doc/whatsnew/3.14.rst). I expect that you would elaborate a bit on the usage, how to use the feature, why it's needed, etc.

stefanor · 2024-08-28T12:04:37Z

You should document this change in https://docs.python.org/dev/whatsnew/3.14.html

Done.

FWIW, in Debian we plan to backport this to 3.13 too.

vstinner · 2024-08-29T09:36:33Z

I'm only aware of Debian who uses "multiarch". Do other operating systems also use it? Maybe Debian variants, Ubuntu, and Ubuntu variants?

This change will slow down any "import module". I don't recall if there is a cache for that or not.

stefanor · 2024-08-29T10:07:37Z

Maybe Debian variants, Ubuntu, and Ubuntu variants?

All Debian derivatives, yes. They typically don't deviate very much, when it comes to plumbing.

stefanor · 2024-08-29T10:33:00Z

This change will slow down any "import module". I don't recall if there is a cache for that or not.

What do you want to do about that? Is there a benchmark you'd like to see results for?

I see a table of 5 entries (including NULL) increasing to 6. That is one extra item to search, when:

Importing C extensions from multiarch filenames.
Importing C extensions with no tag of any kind (foo.so).
Failing to import something because there isn't a C extension with this name.

vstinner · 2024-09-02T13:20:47Z

What do you want to do about that? Is there a benchmark you'd like to see results for?

Example of command:

strace -o trace python3 -c pass

strace output:

newfstatat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__init__.cpython-312-x86_64-linux-gnu.so", 0x7ffe20002e50, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__init__.abi3.so", 0x7ffe20002e50, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__init__.so", 0x7ffe20002e50, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__init__.py", {st_mode=S_IFREG|0644, st_size=5884, ...}, 0) = 0

newfstatat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__init__.py", {st_mode=S_IFREG|0644, st_size=5884, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib64/python3.12/encodings/__pycache__/__init__.cpython-312.pyc", O_RDONLY|O_CLOEXEC) = 3

Internally, when Python imports the encoding package, it has to do 4 fstat() calls. You can see a check on the .abi3.so suffix.

If we add a new entry to the import suffix, it will add one fstat() syscall per import (at least, to import a package). We should consider the impact on performance if we decide to add this feature.

freakboy3742 · 2024-09-02T22:03:15Z

I'm only aware of Debian who uses "multiarch". Do other operating systems also use it? Maybe Debian variants, Ubuntu, and Ubuntu variants?

FWIW: macOS, iOS and Android all use multiarch as a configuration value - it's used to differentiate ARM from x86_64 simulators, and devices from simulators/emulators.

However, on iOS and Android, there's limited need to keep binary artefacts in the same folder, as any given executable can only have a single architecture's executables. In addition, in the case of iOS, the binaries need to be migrated to the Frameworks folder and named as Frameworks, so any "side-by-side" benefits would be lost. Any "other platform" executables need to be stripped out as part of the build process, at which point there's no naming conflict.

macOS uses the multiarch config value, but the value is always "darwin", and the universal binary format exists to support multiple architectures in a single binary file. I guess it might be useful to be able to have x86_64 and ARM64 binaries side-by-side... but there's probably only a year or two left in the official supported life of x86_64, so I don't think adding this feature will ultimately be that helpful for the macOS use case.

vstinner · 2024-09-02T22:25:06Z

@stefanor: Did you consider to maintain this change as a downstream-only patch in Debian?

If you would like to make it upstream, I would suggest making it optional, disabled by default, and add a configure option to enable it. That's how I added some Fedora specific changes, such as:

stefanor · 2024-09-03T23:24:58Z

Here are some benchmarks:

Google Sheet with Data

There is no discernable performance difference in minimal interpreter startup python -c ''.

import time
import sys
from subprocess import check_call

t1 = time.perf_counter()
for i in range(1000):
    check_call([sys.executable, "-c", ""])
t2 = time.perf_counter()
print((t2-t1) / 1000)

Looking at strace, I see only a single extra syscall.

We can manufacture an import-intensive benchmark:

import pkgutil
import time

t1 = time.perf_counter()
for module in pkgutil.walk_packages(onerror=lambda pkg: None):
    if module.name.startswith("test."):
        continue
    if module.name.endswith(".__main__"):
        continue
    if module.name in {"antigravity", "idlelib.idle", "this", "zen"}:
        continue
    try:
        __import__(module)
    except Exception:
        pass
t2 = time.perf_counter()
print((t2 - t1))

Analysing this, I see 116 stat syscalls on .so filenames jumping to 232. But in a 100ms total execution time, that's negligible, and I can't see any statistically significant results.

stefanor · 2024-09-03T23:37:59Z

I'm only aware of Debian who uses "multiarch".

Also, note that this is used on all linux platforms. The default for (non-stable ABI) extensions is to include the multiarch tuple in the extension filename. Pick a random binary wheel built in manylinux, and you'll see them.

@stefanor: Did you consider to maintain this change as a downstream-only patch in Debian?

I would be happy to do that. Although I prefer not carrying downstream-only patches long-term if possible. Look at the mess around dist-packages for example.

Without this MR, there's no real point in supporting the multiarch tags in the non-stable ABI extensions. Either we get the benefit from doing it everywhere, or you say you don't need to support the feature, and we rip it out everywhere. We've got half a feature at the moment. Would you prefer it to all be behind a config argument?

If you would like to make it upstream, I would suggest making it optional, disabled by default, and add a configure option to enable it. That's how I added some Fedora specific changes, such as:

These are not entirely Fedora-specific. And that's a good argument for always trying to upstream.

https://docs.python.org/dev/using/configure.html#cmdoption-with-platlibdir

In Debian we use those paths for our cross-compilers. I imagine there is a scenario where it's useful to have Python headers in there.

https://docs.python.org/dev/using/configure.html#cmdoption-with-wheel-pkg-dir

We're a happy customer of this too, It meant one less patch to carry.

stefanor · 2024-09-04T14:26:19Z

There is no discernable performance difference in minimal interpreter startup python -c ''.

Maybe 0.5% slowdown with PGO. For 1 extra syscall, I'm surprised it's visible. This may still not be significant.

vstinner · 2024-09-18T17:20:26Z

@erlend-aasland @encukou: Would you mind to have a look at this issue? What do you think? Should it become the default behavior, or should it be a configure option?

encukou · 2024-09-23T22:45:00Z

IMO, the most important thing here is to keep this in mind for abi4 (or whatever we name the free-threading-compatible stable ABI). There it should be the only option.
If we get abi4 in 3.14, it's probably fine to keep this as a downstream-only patch for one more release.

stefanor · 2024-09-24T07:30:49Z

IMO, the most important thing here is to keep this in mind for abi4 (or whatever we name the free-threading-compatible stable ABI). There it should be the only option.

Yeah, that sounds sensible.

If we get abi4 in 3.14, it's probably fine to keep this as a downstream-only patch for one more release.

Is that expected? We've been on abi3 for quite a while.

stefanor · 2024-09-24T07:38:03Z

There it should be the only option.

If we want that for the stable ABI, do we want to do the same thing for the regular ABI? It would probably affect a lot of build systems that assume they can name their output extension.so

encukou · 2024-09-24T17:09:47Z

Is that expected? We've been on abi3 for quite a while.

Yes, over a decade. abi3 showing its age, and since it's incompatible free-threading we'll need a new alternative soon.
abi3 is not going away, but changing it this late in the cycle might not be worth it.

If we want that for the stable ABI, do we want to do the same thing for the regular ABI?

I don't see much reason to deprecate and remove that, so that we don't break the build systems you mention. (Note that even abi3/abi4 extensions will work with a bare .so extension, since the stable ABI is a subset of the full one.)

stefanor · 2024-10-07T21:57:13Z

FWIW, in Debian we plan to backport this to 3.13 too.

I'm going to include it in our 3.13.0 upload.

stefanor · 2024-12-19T11:11:37Z

🔔 (I'd like to remind people to please consider this)

stefanor · 2026-06-26T13:16:17Z

Rebased onto main to get the benefit of #150208 for tests.

…ple in the filename This permits stable ABI extensions for multiple architectures to be co-installed into the same directory, without clashing with each other, the same way (non-stable ABI) regular extensions can. It is listed below the current .abi3 suffix because setuptools will select the first suffix containing .abi3, as the target filename. We do this to protect older Python versions predating this patch.

eli-schwartz · 2026-06-26T13:32:54Z

If this is still changed, it really should be done yesterday, and it might arguably already be too late to do the "or" rather than "look for abi3t-powerpc64le-linux-gnu.so first, fall back to abi3t.so". Because build backends and build systems are already released with abi3t support, and doing the unconditional change to ``abi3t-xxx.so` would need new releases of Maturin, scikit-build-core and CMake at this point (extension name is hardcoded there currently I believe).

I'm certain that build backends will be happy to do new releases in order to align with the technical decisions of an evolving situation in unstable, unreleased cpython. This isn't a barrier.

It's unlikely any wheels were already uploaded to PyPI, but it's not impossible - those would have to be yanked or hard-deleted (with lock files, yanking may not be enough).

If they were uploaded then they're already broken by default since the ABI won't be frozen until rc1. Given cpython refuses to limit itself out of concern that wheels with wrong ABI / crashing extensions will have already been uploaded, it would be heavily inconsistent to limit itself out of concern that wheels with undetected / wrong filename extensions might have been uploaded to PyPI.

They're both just as "naughty" / unsupported, but the breakage if someone did the naughty thing is far worse for wrong ABI, and that's already a routine event in cpython development.

vstinner · 2026-06-26T14:40:58Z

doing the unconditional change to ``abi3t-xxx.so` would need new releases of Maturin, scikit-build-core and CMake at this point (extension name is hardcoded there currently I believe).

Again, that's why I think that a discussion on discuss.python.org would be better (to coordinate). Maybe in the Packaging category.

stefanor · 2026-06-26T16:25:40Z

OK: https://discuss.python.org/t/can-we-consider-an-uncoditional-change-of-abi3t-so-to-abi3t-soabi-platform-so-for-3-15-last-minute/107919

…uple in the filename This permits `abi3` stable ABI extensions for multiple architectures to be co-installed into the same directory, without clashing with each other, the same way (non-stable ABI) regular extensions can. It is listed before the current .abi3 suffix because .abi3 extensions are only compatible with future Python releases, not older versions. This follows on from python#122917 which was reduced down to just targetting `abi3t`

stefanor · 2026-06-27T22:59:35Z

Now that this PR is just renaming abi3t there is a follow-on commit in #152461 that adds tags to abi3 (the original aim of this PR)

…ple in the filename This permits stable ABI extensions for multiple architectures to be co-installed into the same directory, without clashing with each other, the same way (non-stable ABI) regular extensions can. It is listed below the current .abi3 suffix because setuptools will select the first suffix containing .abi3, as the target filename. We do this to protect older Python versions predating this patch.

stefanor · 2026-07-18T14:41:55Z

Updated to reflect the SC request for both .abi3t.so and .abi3t-TAG.so

vstinner · 2026-07-18T14:46:37Z

@@ -0,0 +1 @@
+Linux: Include multiarch tuples in ``abi3t`` stable ABI C extension filenames, e.g. ``foo.abi3t-x86-64-linux-gnu.so``, by default.


"Linux: ": this change doesn't seem to be specific to Linux, is it?

Those tags won't exist on not-Linux

Maybe we should change the text to say on platforms with multiarch tags (e.g. Linux).

vstinner

The current implementation doesn't work properly on FreeBSD, because the SOABI_PLATFORM macro is defined but it's defined to an empty string.

$ grep SOABI_PLATFORM pyconfig.h
#define SOABI_PLATFORM ""

In Python:

>>> import importlib.machinery
>>> importlib.machinery.EXTENSION_SUFFIXES
['.cpython-316d.so', '.cpython-316.so', '.abi3.so', '.abi3t-.so', '.abi3t.so', '.so']
>>> import sysconfig
>>> SOABI_PLATFORM = sysconfig.get_config_var("SOABI_PLATFORM")
>>> SOABI_PLATFORM
''

Notice that '.abi3t-.so' is present in EXTENSION_SUFFIXES.

Possible fix:

diff --git a/configure.ac b/configure.ac
index 2d6672e3231..c30d98af28a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1212,7 +1212,9 @@ AS_CASE([$ac_sys_system],
   [SOABI_PLATFORM=$PLATFORM_TRIPLET]
 )
 
-AC_DEFINE_UNQUOTED([SOABI_PLATFORM], ["${SOABI_PLATFORM}"], [Platform tag, used in binary module extension filenames.])
+if test x$SOABI_PLATFORM != x; then
+    AC_DEFINE_UNQUOTED([SOABI_PLATFORM], ["${SOABI_PLATFORM}"], [Platform tag, used in binary module extension filenames.])
+fi
 
 if test x$MULTIARCH != x; then
   MULTIARCH_CPPFLAGS="-DMULTIARCH=\\\"$MULTIARCH\\\""

Add ALT_SOABI and SOABI_PLATFORM to test.pythoninfo. Add also EXTENSION_SUFFIXES of importlib.machinery.

vstinner · 2026-07-18T16:46:28Z

I pushed a change in your branch to log ALT_SOABI, SOABI_PLATFORM and EXTENSION_SUFFIXES in test.pythoninfo.

vstinner · 2026-07-18T16:52:14Z

For example, I looked at the WASI job.

Extracts of test.pythoninfo on WASI:

importlib.extension_suffixes: ['.cpython-316d-wasm32-wasi.so', '.cpython-316-wasm32-wasi.so', '.abi3.so', '.abi3t-wasm32-wasi.so', '.abi3t.so', '.so']

sysconfig[ABIFLAGS]: d
sysconfig[ALT_SOABI]: cpython-316-wasm32-wasi
sysconfig[SOABI]: cpython-316d-wasm32-wasi
sysconfig[SOABI_PLATFORM]: wasm32-wasi

@stefanor:

Those tags won't exist on not-Linux

This changes adds '.abi3t-wasm32-wasi.so' to EXTENSION_SUFFIXES on WASI. Is it deliberate?

vstinner · 2026-07-18T16:55:06Z

pythoninfo on macOS:

importlib.extension_suffixes: ['.cpython-316d-darwin.so', '.cpython-316-darwin.so', '.abi3.so', '.abi3t-darwin.so', '.abi3t.so', '.so']
sysconfig[SOABI_PLATFORM]: darwin

The PR adds .abi3t-darwin.so suffix.

pythoninfo on iOS:

importlib.extension_suffixes: ['.cpython-316-iphonesimulator.so', '.abi3.so', '.abi3t-iphonesimulator.so', '.abi3t.so', '.so']

The PR adds .abi3t-iphonesimulator.so suffix.

stefanor · 2026-07-18T19:19:44Z

Thanks for checking that Victor

Notice that '.abi3t-.so' is present in EXTENSION_SUFFIXES.

Agreed that that's not what we want.

This changes adds '.abi3t-wasm32-wasi.so' to EXTENSION_SUFFIXES on WASI. Is it deliberate?

I don't know much about the WASI stack, but that's probably a reasonable thing to do if one is likely to have WASI builds near native builds.

I'll add your configure change, it seems reasonable.

stefanor · 2026-07-18T19:59:58Z

FWIW, it looks like #152461 (which was originally stacked on top of this) may merge, including these changes and more.

stefanor requested review from ericsnowcurrently and erlend-aasland as code owners August 11, 2024 21:32

bedevere-app Bot added the awaiting review label Aug 11, 2024

stefanor force-pushed the stable-abi-multiarch branch 2 times, most recently from 5e380a1 to 7185a58 Compare August 12, 2024 07:35

stefanor force-pushed the stable-abi-multiarch branch from 7185a58 to 2955027 Compare August 12, 2024 07:51

stefanor changed the title ~~Allow stable API extensions to include a multiarch tuple in the filename~~ gh-122931 Allow stable API extensions to include a multiarch tuple in the filename Aug 12, 2024

bedevere-app Bot mentioned this pull request Aug 12, 2024

Allow stable ABI extensions to include multiarch tuples in the filename #122931

Open

stefanor force-pushed the stable-abi-multiarch branch from 2955027 to c5b58a4 Compare August 12, 2024 09:21

vstinner reviewed Aug 28, 2024

View reviewed changes

stefanor requested a review from encukou as a code owner August 28, 2024 12:03

stefanor force-pushed the stable-abi-multiarch branch from f84ef65 to 67e2548 Compare June 26, 2026 13:15

stefanor changed the title ~~gh-122931: Allow stable API extensions to include a multiarch tuple in the filename~~ gh-122931: Include multiarch tuple in abi3t filenames Jun 27, 2026

stefanor mentioned this pull request Jun 27, 2026

gh-122931: Allow stable abi3 API extensions to include a multiarch tuple in the filename #152461

Open

stefanor force-pushed the stable-abi-multiarch branch from 67e2548 to 238f194 Compare July 18, 2026 14:29

stefanor added 7 commits July 18, 2026 11:32

Order the multiarch-tagged extensions first

8d56569

Test for presence of multiarch extension suffixes

f5d7367

Fix editing error in NEWS and improve clarity

e79e570

Fix up the test

46e746d

Only target abi3t (for expediency)

47116a4

Keep .abi3t.so, as requested by the SC

b4a6389

stefanor force-pushed the stable-abi-multiarch branch from 238f194 to b4a6389 Compare July 18, 2026 14:40

vstinner reviewed Jul 18, 2026

View reviewed changes

Add SOABI_PLATFORM to test.pythoninfo

71ff855

Add ALT_SOABI and SOABI_PLATFORM to test.pythoninfo. Add also EXTENSION_SUFFIXES of importlib.machinery.

Only define SOABI_PLATFORM if it is set

5736916

		@@ -0,0 +1 @@
		Linux: Include multiarch tuples in ``abi3t`` stable ABI C extension filenames, e.g. ``foo.abi3t-x86-64-linux-gnu.so``, by default.

Uh oh!

Uh oh!

Conversation

stefanor commented Aug 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

stefanor commented Aug 28, 2024

Uh oh!

vstinner commented Aug 29, 2024

Uh oh!

stefanor commented Aug 29, 2024

Uh oh!

stefanor commented Aug 29, 2024

Uh oh!

vstinner commented Sep 2, 2024

Uh oh!

freakboy3742 commented Sep 2, 2024

Uh oh!

vstinner commented Sep 2, 2024

Uh oh!

stefanor commented Sep 3, 2024

Uh oh!

stefanor commented Sep 3, 2024

Uh oh!

stefanor commented Sep 4, 2024

Uh oh!

vstinner commented Sep 18, 2024

Uh oh!

encukou commented Sep 23, 2024

Uh oh!

stefanor commented Sep 24, 2024

Uh oh!

stefanor commented Sep 24, 2024

Uh oh!

encukou commented Sep 24, 2024

Uh oh!

stefanor commented Oct 7, 2024

Uh oh!

stefanor commented Dec 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stefanor commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eli-schwartz commented Jun 26, 2026

Uh oh!

vstinner commented Jun 26, 2026

Uh oh!

stefanor commented Jun 26, 2026

Uh oh!

stefanor commented Jun 27, 2026

Uh oh!

stefanor commented Jul 18, 2026

Uh oh!

vstinner Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

stefanor Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

stefanor Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

vstinner left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vstinner commented Jul 18, 2026

Uh oh!

vstinner commented Jul 18, 2026

Uh oh!

vstinner commented Jul 18, 2026

Uh oh!

stefanor commented Jul 18, 2026

Uh oh!

stefanor commented Jul 18, 2026

Uh oh!

stefanor commented Aug 11, 2024 •

edited

Loading

stefanor commented Dec 19, 2024 •

edited

Loading

stefanor commented Jun 26, 2026 •

edited

Loading

vstinner left a comment •

edited

Loading