fix: delete coglet-managed output files after upload

[anish-sahoo]

Problem

Coglet creates output files (IOBase writes, oversized JSON spills) in /tmp/coglet/predictions/{id}/outputs/ but never deletes them after upload or base64 encoding. This can cause stale-output bugs where models return outputs from previous predictions if an overwrite fails.

Closes #1434.

Real-life examples

1. Fixed-path output with silent overwrite failure
A model writes its output to a fixed temp path (e.g. /tmp/output.png) and returns that Path. On a subsequent prediction, the image generation step hits an OOM or CUDA error that is caught by a try/except, but the model still returns the same path. Since the previous prediction's file was never cleaned up, the orchestrator reads and uploads the stale image — the user receives output from the previous prediction without any error.

2. Partial failure in multi-output models
A model generates multiple output files and returns them as a list of Path objects. If one of the file writes fails (e.g. disk full, permission error on a specific subdirectory), the model may still return all paths. The orchestrator uploads the stale file from the previous prediction for the failed write, producing a mixed output: some files from the current run, some from the previous run.

3. Disk exhaustion from accumulated outputs
In high-throughput deployments, output files accumulate in /tmp/coglet/predictions/ across hundreds of predictions. Large outputs (video files, high-res images) can fill the container's tmpfs, causing subsequent predictions to fail with No space left on device errors. This is a slow-burn operational issue that degrades reliability over time until the container is restarted.

4. Cancelled prediction leaves orphaned files
When a prediction is cancelled (e.g. client disconnects), in-flight upload tasks are aborted and their output files are left on disk. These orphans accumulate over the container's lifetime and are never cleaned up, contributing to the disk exhaustion problem above.

Solution

Add a managed: bool field to the FileOutput IPC protocol message so the orchestrator knows which files are safe to delete (coglet-created) vs user-authored Path outputs (returned by reference, must not be deleted).

• Orchestrator: deletes managed files immediately after reading their bytes into memory (covers all three branches: Oversized, upload, base64)
• Backstop: remove_prediction now does remove_dir_all on the prediction directory to catch files from aborted uploads or cancelled predictions
• User Path outputs: marked managed: false, never deleted

Files changed

• crates/coglet/src/bridge/protocol.rs — added managed: bool field with #[serde(default)]; 3 snapshot tests
• crates/coglet/src/worker.rs — send_file_output gains managed param; write_file_output and build_output_message set true
• crates/coglet-python/src/predictor.rs — PathLike call sites pass false
• crates/coglet/src/orchestrator.rs — deletes managed files after std::fs::read
• crates/coglet/src/service.rs — prediction_dir_for helper; remove_dir_all backstop in remove_prediction; 2 tests

Validation

• mise run test:rust — 220/220 pass
• mise run lint:rust — clippy clean
• mise run fmt:rust — formatting clean

[ask-bonk]

LGTM

github run