Skip to content

feat(live): persist media frames alongside audio chunks in save_live_blob - #7062

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
copybara/977664760
Open

feat(live): persist media frames alongside audio chunks in save_live_blob#7062
copybara-service[bot] wants to merge 1 commit into
mainfrom
copybara/977664760

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Sep 8, 2026

Copy link
Copy Markdown

feat(live): persist media frames alongside audio chunks in save_live_blob

Previously, save_live_blob=True on RunConfig only persisted audio streams via AudioCacheManager and AudioCacheConfig. Any video or media frames streamed during live sessions (e.g. camera feeds or screen captures) were dropped and unrecoverable from session artifacts.

This change upgrades the persistence subsystem to support arbitrary media frames alongside audio chunks, replacing AudioCacheManager with a unified CacheManager architecture.

Key Architectural Changes:

  1. Unified Cache Engine (live/_cache_manager.py):

    • Replaces AudioCacheManager and AudioCacheConfig with CacheManager and CacheConfig.
    • Adds dual caching queues (audio chunks and media frames) for both input and output streams.
    • Implements MIME-type routing (audio/* vs image/*/video/*).
    • Implements FIFO eviction policies enforcing both frame count limits (max_media_cache_frames=600) and byte limits (max_media_cache_size_bytes=100MB).
    • Supports concurrent cache flushing across all active channels via asyncio.gather(..., return_exceptions=True).
  2. Artifact Persistence (artifacts/):

    • Adds concrete save_media_frames() to BaseArtifactService (raising NotImplementedError by default to maintain backward compatibility with custom external implementations) and implements it across FileArtifactService, InMemoryArtifactService, GcsArtifactService, _ForwardingArtifactService, and PerAgentFileArtifactService.
    • Layout: writes individual frames as frames/frame_{index:04d}.{ext}, generates a structured metadata.json (timestamps, FPS, duration, frame list), and links frame_0000 for preview compatibility with load_artifact().
    • Uses atomic file writes for directory creation and metadata updates.
  3. Flow Integration (flows/llm_flows/):

    • Updates BaseLlmFlow and _LiveLlmFlow to route incoming and outgoing media blobs to the appropriate cache queues.
    • Handles 4-channel concurrent flush on session teardown or turn completion (input audio, input media, output audio, output media).
  4. Backward Compatibility:

    • AudioCacheManager and AudioCacheConfig are replaced by CacheManager and CacheConfig.
    • Maintained a deprecated audio_cache_manager property on BaseLlmFlow returning self.cache_manager for backward compatibility with callers and public API checks.
  5. Documentation:

    • Added CacheManager developer guide in docs/guides/live/cache_manager/ and updated developer guides in docs/guides/artifacts/artifact_service/ and docs/guides/live/live_request_queue/ with multimodal persistence and streaming examples.

Testing:

  • Added comprehensive unit tests in tests/unittests/live/test_cache_manager.py (24 tests covering model validation, configuration, blob routing, FIFO eviction, and concurrent flushes).
  • Added multi-backend artifact tests in test_artifact_service.py (file, in-memory, GCS).
  • Verified test_live_llm_flow.py, test_base_llm_flow_realtime.py, test_runners.py, and test_forwarding_artifact_service.py.
  • Verified local storage and CLI tests in test_local_storage.py, test_service_factory.py, test_cli.py, and test_adk_web_server_tests.py.
  • Full subpackage test suite passing.
  • All precommit formatters and linters passed cleanly (run_precommit.sh).

@copybara-service
copybara-service Bot force-pushed the copybara/977664760 branch 3 times, most recently from 97b682c to cc51c25 Compare September 8, 2026 22:25
…blob

Previously, `save_live_blob=True` on `RunConfig` only persisted audio streams via `AudioCacheManager` and `AudioCacheConfig`. Any video or media frames streamed during live sessions (e.g. camera feeds or screen captures) were dropped and unrecoverable from session artifacts.

This change upgrades the persistence subsystem to support arbitrary media frames alongside audio chunks, replacing `AudioCacheManager` with a unified `CacheManager` architecture.

Key Architectural Changes:
1. Unified Cache Engine (`live/_cache_manager.py`):
   - Replaces `AudioCacheManager` and `AudioCacheConfig` with `CacheManager` and `CacheConfig`.
   - Adds dual caching queues (audio chunks and media frames) for both input and output streams.
   - Implements MIME-type routing (`audio/*` vs `image/*`/`video/*`).
   - Implements FIFO eviction policies enforcing both frame count limits (`max_media_cache_frames=600`) and byte limits (`max_media_cache_size_bytes=100MB`).
   - Supports concurrent cache flushing across all active channels via `asyncio.gather(..., return_exceptions=True)`.

2. Artifact Persistence (`artifacts/`):
   - Adds concrete `save_media_frames()` to `BaseArtifactService` (raising `NotImplementedError` by default to maintain backward compatibility with custom external implementations) and implements it across `FileArtifactService`, `InMemoryArtifactService`, `GcsArtifactService`, `_ForwardingArtifactService`, and `PerAgentFileArtifactService`.
   - Layout: writes individual frames as `frames/frame_{index:04d}.{ext}`, generates a structured `metadata.json` (timestamps, FPS, duration, frame list), and links `frame_0000` for preview compatibility with `load_artifact()`.
   - Uses atomic file writes for directory creation and metadata updates.

3. Flow Integration (`flows/llm_flows/`):
   - Updates `BaseLlmFlow` and `_LiveLlmFlow` to route incoming and outgoing media blobs to the appropriate cache queues.
   - Handles 4-channel concurrent flush on session teardown or turn completion (input audio, input media, output audio, output media).

4. Backward Compatibility:
   - `AudioCacheManager` and `AudioCacheConfig` are replaced by `CacheManager` and `CacheConfig`.
   - Maintained a deprecated `audio_cache_manager` property on `BaseLlmFlow` returning `self.cache_manager` for backward compatibility with callers and public API checks.

5. Documentation:
   - Added `CacheManager` developer guide in `docs/guides/live/cache_manager/` and updated developer guides in `docs/guides/artifacts/artifact_service/` and `docs/guides/live/live_request_queue/` with multimodal persistence and streaming examples.

Testing:
- Added comprehensive unit tests in `tests/unittests/live/test_cache_manager.py` (24 tests covering model validation, configuration, blob routing, FIFO eviction, and concurrent flushes).
- Added multi-backend artifact tests in `test_artifact_service.py` (file, in-memory, GCS).
- Verified `test_live_llm_flow.py`, `test_base_llm_flow_realtime.py`, `test_runners.py`, and `test_forwarding_artifact_service.py`.
- Verified local storage and CLI tests in `test_local_storage.py`, `test_service_factory.py`, `test_cli.py`, and `test_adk_web_server_tests.py`.
- Full subpackage test suite passing.
- All precommit formatters and linters passed cleanly (`run_precommit.sh`).
PiperOrigin-RevId: 977664760
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant