feat: implement native PEP 0810 lazy loading

[hebaalazzeh]

Overview

This is an initiative to resolve severe initialization bottlenecks (~10s-13s) in generated client libraries by adopting Native Python 3.15 Explicit Lazy Imports (PEP 0810).
To avoid introducing maintenance burdens or subtle breaking changes via custom import hooks, this PR focuses exclusively on utilizing native interpreter standards.

Implementation Architecture

We modify the upstream GAPIC Generator to emit a native __lazy_modules__ set at the top of the __init__.py files, immediately followed by the standard eager imports.

• Python 3.15+ (Native Acceleration): The Python 3.15 runtime natively intercepts the standard imports referenced in the __lazy_modules__ set and treats them as fast C-level lazy proxies. Startup times drop from ~13s to ~200ms, and peak RAM consumption drops by up to 85%.
• Python 3.14 and Older (Zero Blast Radius): Older Python environments safely ignore the __lazy_modules__ variable and process the standard eager imports exactly as they do today. This guarantees 100% backwards compatibility with zero risk.
• Perfect Static Typing: Because standard imports are still present in the file, IDEs (IntelliSense) and static analyzers (MyPy) maintain zero-friction support.

Example Structure:

# 1. Native C-Level Lazy Proxying for Python 3.15+ (PEP 0810)
__lazy_modules__ = {
   f"{__name__}.services.accelerator_types",
   f"{__name__}.services.addresses",
   f"{__name__}.types.compute",
}
# 2. Standard eager imports
# Evaluated instantly by older Python versions. Intercepted natively by Python 3.15.
from .services.accelerator_types import AcceleratorTypesClient
from .services.addresses import AddressesClient
from .types.compute import Address
__all__ = (
    "AcceleratorTypesClient",
    "AddressesClient",
    "Address",
)

🚀 Performance Benchmarking Results (Disk-Level Cold Start)

We utilized our custom profiler.py script #17467 to run 5 iterations measuring true disk-level cold starts (with __pycache__ sweeps) on the google-cloud-compute library. The results validate this Phase 1 PEP 810 Native Lazy Loading implementation:

Python 3.14 (Current Eager Import)

• Time (Median): 23,246.97 ms (~23.2 seconds)
• Memory (Physical RSS): 224.87 MB
• Code Volume: Loaded 1,407 modules (1,040,992 lines of code)

Python 3.15.0b2 (Phase 1 PEP 810 Lazy Proxy)

• Time (Median): 1,062.87 ms (~1.06 seconds)
• Memory (Physical RSS): 7.21 MB
• Code Volume: Loaded 15 modules (8,064 lines of code)

Impact: By natively deferring the 120+ submodules at the C-level, we bypassed parsing over 1 million lines of code. This resulted in an approximately 22x reduction in startup latency (~95% speedup), and a staggering 96% reduction in peak RAM consumption (dropping from ~225MB down to just ~7MB).

Related Links:
see #17594 for a demonstration of this generator change applied to google-cloud-compute package.
Design Doc: go/sdk-performance-design
Towards googleapis/python-aiplatform#4749

[gemini-code-assist]

Code Review

This pull request adds a conditional check for Python 3.15+ to define __lazy_modules__ in the initialization file of google-cloud-compute. However, the sys module is not imported, which will cause a NameError at runtime. Since this is a generated file, the generator or templates should be updated to include the necessary import rather than applying manual fixes.

[hebaalazzeh]

/gemini