Skip to content

Add file and screenshot input for local agents - #39

Open
lvwerra wants to merge 4 commits into
mainfrom
feat/screenshot-input
Open

Add file and screenshot input for local agents#39
lvwerra wants to merge 4 commits into
mainfrom
feat/screenshot-input

Conversation

@lvwerra

@lvwerra lvwerra commented Aug 5, 2026

Copy link
Copy Markdown
Member

Summary

  • accept up to five regular files per turn at 100 MiB each, including PDF, DOCX and other Office files, archives, source files, and unknown formats
  • keep validated PNG, JPEG, GIF, and WebP files on the native image path while delivering other files to the agent by their managed session-scoped paths
  • support paste, drop, and file picking in active sessions; keep the quick-creation dialog uncluttered by omitting its redundant file picker while retaining paste and drop
  • use one structured, race-safe delivery path with image-only/file-only turns, native first-turn Codex images, Hermes image commands, and retryable terminal insertion
  • prevent duplicate clipboard images by treating the canonical clipboard file list as the single source for synchronous paste events
  • stream uploads with per-file and per-session limits, sanitize bounded filenames, detect raster formats from bytes, and force non-raster files such as SVG and HTML to download instead of rendering inline
  • retain attachments through stopped-session retries, make insertion idempotent, lock chips for the full send transaction, and protect shared viewers from injection

Remote agents remain explicitly disabled because they cannot read files stored on the Space.

Limits

  • 100 MiB per file
  • 5 files per turn
  • 500 MiB and 200 files per session

Verification

  • npm test in server
  • node test/attachments.test.mjs in server
  • npm run typecheck in web
  • npm run test:screenshots in server
  • node terminal-ui.test.mjs in server
  • git diff --cached --check

Design

The architecture, security model, failure behavior, limits, and manual PDF/DOCX test matrix are documented in docs/screenshot-input.md.

@lvwerra lvwerra changed the title Add screenshot input for local agents Add file and screenshot input for local agents Aug 6, 2026
@lvwerra lvwerra closed this Aug 6, 2026
@lvwerra lvwerra reopened this Aug 6, 2026
@lvwerra
lvwerra marked this pull request as ready for review August 6, 2026 22:06
lvwerra pushed a commit that referenced this pull request Aug 7, 2026
Six things from reading a real session in a real pane.

**Zoom reaches the reader.** It covers a terminal whose font the zoom keys
scale, so a reader that ignored them made the zoom control look broken. The
conversation and the reply line scale; the toolbar and the path footer are
chrome and stay put.

**The prompt band spans the pane.** It reached into the left gutter but stopped
at the text column on the right, which is what made it read as a card floating
over the answer rather than as the head of it. Full bleed both sides, the meta
row tight under the band it belongs to, and daylight between exchanges.

**Everything opens on the newest turn.** Reader mode landed at the top of a
400-turn window — scroll through a month to reach the thing you flipped the
switch to read. The trace viewer was worse: it opened on page 0. Both land on
the last turn now; the viewer does it by the same two-step the prompt nav uses,
because row heights start as estimates.

**The bottom bar is one height.** The mode toggle was 19px next to 22px keys.

**Nothing of the terminal's paints over the reader.** Its covers — `restoring
last view…`, `starting claude…`, `stopped · output preserved` — are z-index 4
and the overlay was 3, so a reconnect drew the last terminal screen straight
over the conversation, leaving a reader toolbar above a terminal. They are
gated off while reading, and the overlay outranks anything the terminal can
raise. Reported from dev-3; reproduced by injecting the real cover markup.

**One composer, and it is honestly optimistic.** The card and the reader render
the same `Composer` — duplicated markup is how one surface quietly gets a
feature and the other does not. The seam for PR #39's screenshot input is
`onPasteFiles` + `above`; until #39 lands neither surface accepts a pasted
image, and wiring both is then one place. While extracting it: the echo was not
actually optimistic — it waited for the POST, leaving a beat where the box was
still full and nothing had happened. It now goes up first and is withdrawn,
text restored, if the send fails.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant