Skip to content

fix(proxy): reset keyset and mapping settings on DISCARD ALL and RESET (2.2.x, CIP-5021) - #475

Merged
tobyhede merged 8 commits into
release/2.2from
claude/project-thread-gvoja6-2.2
Oct 21, 2026
Merged

tobyhede merged 8 commits into
release/2.2from
claude/project-thread-gvoja6-2.2

Conversation

@tobyhede

@tobyhede tobyhede commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Backport of CIP-5021 to the 2.2 line. Stacked on #473 (the 2.2 connection-state leak fix); the main-branch fix is #474.

Before: a connection pool that reset a connection between clients with DISCARD ALL or RESET ALL handed the next client the previous client's keyset, and left CIPHERSTASH.UNSAFE_DISABLE_MAPPING on, so the next client's queries were sent unencrypted. Proxy rejected every RESET statement as invalid SQL. SET CIPHERSTASH.<setting> TO DEFAULT left the setting as it was, or for KEYSET_NAME selected a keyset named "DEFAULT".

After: when PostgreSQL confirms DISCARD ALL or RESET ALL, Proxy clears the session keyset and turns UNSAFE_DISABLE_MAPPING off. Proxy then uses default_keyset_id, or requires a keyset again when there is none. RESET CIPHERSTASH.KEYSET_ID / KEYSET_NAME / UNSAFE_DISABLE_MAPPING and SET CIPHERSTASH.<setting> TO DEFAULT reset only that setting. Names match case-insensitively, including a quoted name with the period. A reset that PostgreSQL rejects, for example in an aborted transaction, changes nothing. Both the simple and the extended protocol are covered.

How

  • Resets reuse fix(proxy): release per-connection protocol state when PostgreSQL is done with it (2.2.x, CIP-5015) #473's batch mechanism for SQL deallocations. Each reset is queued with the generation of the batch it was sent in. It is applied on the matching RESET / SET / DISCARD ALL CommandComplete. If PostgreSQL never confirms it, the batch's ReadyForQuery discards it.
  • A simple Query queues one entry per SET / DISCARD ALL statement, so each tag lines up with its own statement. It queues nothing when none of them resets a Proxy setting.
  • A prepared SET / RESET / DISCARD ALL records its reset on the statement. Bind carries it onto the portal, and every Execute queues it.
    • On 2.2 this includes SETs that reset nothing. Main tracks each operation separately, but 2.2 has one queue per connection, so this keeps tags in step when a batch has several SETs.
  • The bundled parser has no RESET statement. SqlParser::parse_resets recognises a query made only of RESET statements, and Proxy calls it only after the normal parse fails.
  • unsafe_disable_mapping is now an Arc<AtomicBool> shared by the frontend and backend Context clones.
  • Per-setting change counts stop a late reset from undoing a SET CIPHERSTASH.* the client sent after the reset.
    • On 2.2, the counts are updated under a mutex, together with the setting itself.
  • The schema-intent cleanup for prepared RESETs from fix(proxy): reset keyset and mapping settings on DISCARD ALL and RESET (CIP-5021) #474 is not ported, because 2.2 has no schema middleware.
  • Tests:
    • New unit tests in postgresql/tests.rs, plus parser and command-tag tests. They drive a real Frontend and Backend over in-memory streams with wire messages.
    • New integration tests pool_reset_turns_mapping_back_on and pool_reset_clears_keyset_id.

Acknowledgment

By submitting this pull request, I confirm that CipherStash can use, modify, copy, and redistribute this contribution, under the terms of CipherStash's choice.

Generated by Claude Code

@tobyhede tobyhede self-assigned this Oct 19, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T05:50:10.931479Z ccd6e39 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ccd6e39b7e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/cipherstash-proxy/src/postgresql/context/mod.rs
Comment thread packages/cipherstash-proxy/src/postgresql/context/mod.rs
@tobyhede
tobyhede force-pushed the claude/project-thread-gvoja6-2.2 branch from 2077279 to ff6d54a Compare October 19, 2026 05:59
Base automatically changed from fix/2.2-connection-state-leak to release/2.2 October 19, 2026 06:34
…T (CIP-5021)

A connection pool that resets a connection between clients with
DISCARD ALL or RESET ALL handed the next client the previous client's
keyset, and left UNSAFE_DISABLE_MAPPING on so its queries were sent
unencrypted.

When PostgreSQL confirms DISCARD ALL or RESET ALL, Proxy now clears the
session keyset (falling back to default_keyset_id) and turns
UNSAFE_DISABLE_MAPPING off. RESET CIPHERSTASH.<setting> and
SET CIPHERSTASH.<setting> TO DEFAULT reset just that setting, and
SET CIPHERSTASH.KEYSET_NAME TO DEFAULT no longer selects a keyset named
"DEFAULT".

Resets use the same batch mechanism as SQL deallocations: each one is
queued with the generation of the batch it was sent in, applied on the
matching RESET, SET or DISCARD ALL CommandComplete, and discarded by the
batch's ReadyForQuery if PostgreSQL rejected it. A simple Query queues
one entry per SET or DISCARD ALL when any of them resets a Proxy
setting; a prepared SET, RESET or DISCARD ALL carries its reset onto the
portal and queues it on each Execute, including SETs that reset
nothing, so each tag in a pipelined batch lines up with its statement.
Per-setting change counts keep a late reset from undoing a SET the
client sent after it.

The bundled SQL parser has no RESET statement, so Proxy rejected every
RESET as invalid SQL. SqlParser::parse_resets recognises a query made
only of RESET statements and is tried only when the normal parse fails.

UNSAFE_DISABLE_MAPPING is now shared by the frontend and backend
Context clones, so the backend can reset it.
… on (CIP-5021)

Cover DISCARD ALL, RESET ALL, RESET CIPHERSTASH.<setting> and
SET CIPHERSTASH.<setting> TO DEFAULT through both the simple and the
extended protocol, and add the changelog entry.
…s (CIP-5021)

Proxy applied SET CIPHERSTASH.KEYSET_ID, KEYSET_NAME and
UNSAFE_DISABLE_MAPPING once, when the statement was parsed. A named
prepared SET that ran again after RESET ALL left Proxy reset while
PostgreSQL had the setting again, so encrypted queries could use no
keyset or the default keyset instead of the one the client selected.

Record each prepared SET CIPHERSTASH statement and apply it again on
every Bind that Proxy accepts, with mapping enabled or disabled. The
change is counted under the settings lock like any other SET, so a
reset still pending at the time does not undo it. The record is removed
with the statement by Close, DEALLOCATE or DISCARD ALL.

@freshtonic freshtonic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design is right. Resets apply only after PostgreSQL confirms them, they are scoped by generation, and the per-setting change counts stop a late reset from undoing a later SET. I mutated each guarantee, and in each case a test failed. But I found one fail-open path that I think must be fixed before merge (finding 1).

I also checked the blocking findings from my #473 review on the current release/2.2. B1 (same-batch DEALLOCATE), B2 (startup ReadyForQuery), N3 (COPY Syncs) and N5 (ErrorResponse pops the next batch's Execute) are all fixed. When I broke each fix, an existing test failed. Thanks.

Please fix before merge

  1. Fail-open: a reset is lost when mapping is disabled and the previous batch's ReadyForQuery arrives between Bind and Execute. (frontend.rs:1088-1100, context/mod.rs:914-928)
    • When mapping is disabled, the Bind early-return path records portal_resets but does not call add_portal.
    • At ReadyForQuery I, discard_portals_before removes every portal_resets entry whose name is not in portals. This includes the portal just bound for the next batch, so the Execute queues nothing.
    • Scenario (I confirmed it with a throwaway unit test):
      1. Prepare r = RESET ALL. Set a keyset and disable mapping.
      2. The client sends Query("SELECT 1"), then Bind("", "r").
      3. The server sends CommandComplete and ReadyForQuery I.
      4. The client sends Execute("") and Sync. The server sends BindComplete, CommandComplete("RESET") and ReadyForQuery.
    • Result: unsafe_disable_mapping() is still true and the keyset is still set, but PostgreSQL has reset both. The next client's queries go to PostgreSQL unencrypted, which is the leak that this PR closes.
    • Suggested fix: on the mapping-disabled Bind path, also call add_portal(bind.portal, Portal::passthrough(session_id)). Better, prune portal_resets and portal_deallocations by the generation of each entry, not by membership of portals. Please add the scenario as a regression test. The same pruning also drops portal_deallocations, but that only leaks a statement.

Non-blocking

  1. In a simple Query, a SET … TO DEFAULT undoes a SET that comes after it in the same Query. (frontend.rs:206)

    • add_settings_resets runs after query_handler, so the counts are read after the loop at :463-479 has applied every SET CIPHERSTASH.*.
    • With keyset k1 set, SET CIPHERSTASH.KEYSET_ID TO DEFAULT; SET CIPHERSTASH.KEYSET_ID = 'k2' gives k2 in PostgreSQL, but None in Proxy. Proxy then uses default_keyset_id or fails. I confirmed this with a throwaway test.
    • Main (#474) reads the counts before the loop (middleware/frontend.rs:425). That makes this order correct on main but the reverse order wrong: SET …UNSAFE_DISABLE_MAPPING = true; SET …UNSAFE_DISABLE_MAPPING TO DEFAULT leaves mapping disabled after PostgreSQL resets it, which fails open. I found this by reading the code and did not run it.
    • Suggested fix for both branches: read the counts for each reset at its position in the loop, queue the entries after the loop, and test both orders.
  2. A keyset SET sent before a mapping reset is confirmed is ignored. (frontend.rs:472-479, :841, :961)

    • While mapping is disabled, Proxy skips handle_set_keyset. A RESET or TO DEFAULT now turns mapping on only at confirmation, so a keyset SET pipelined before then is dropped.
    • With keyset k1 set and mapping disabled, pipeline Query("RESET CIPHERSTASH.UNSAFE_DISABLE_MAPPING") and Query("SET CIPHERSTASH.KEYSET_ID = 'k2'"). Mapping turns on, but Proxy still has k1 while PostgreSQL has k2. I confirmed this with a throwaway test.
    • Suggested fix: apply keyset SETs while mapping is disabled, because they only record a setting. At a minimum, record them while a mapping reset is pending.
  3. Nit: add_settings_resets (context/mod.rs:1027-1033) returns early if the mutex is poisoned. Then it drops the entry and the batch's tags go out of step. This needs a panic first, so it is theoretical. Please queue the entry anyway.

Codex comments

  • "Reapply prepared SET effects on every execution": 956c9b3 fixes this, and I verified it.
  • "Restore proxy settings when reset transactions roll back": I agree that it can be a follow-up (CIP-5058). Please record in CIP-5058 that a rolled-back RESET now clears the keyset, so Proxy uses default_keyset_id while PostgreSQL still has the tenant keyset.

Out of scope (older code, found by reading, not run): in query_handler, when any statement in a Query is rewritten, the forwarded SQL is rebuilt only from transformed_statements. So a SET or DISCARD that took a continue (:472-483) is not sent to PostgreSQL. For this PR, a reset in a Query that also has an encrypted INSERT never reaches PostgreSQL, but Proxy discards its entry, so the two stay consistent. Please consider a separate ticket for this.

Address review of the 2.2 backport:

- A reset bound for the next batch while PostgreSQL was still answering
  the previous one was lost. Completing the earlier execution closed the
  unnamed portal and removed the newer Bind's reset, and with mapping
  disabled the ReadyForQuery cleanup also dropped it because the Bind
  adds no portal. Portal effects now carry the generation they were
  bound in, are pruned by generation, and an execution only removes the
  effects bound at or before it.
- In a simple Query, each reset now records the settings change counts
  at its own position, so it undoes the SETs before it but not a SET
  after it in the same Query.
- A keyset SET sent while mapping is disabled is recorded when no
  default keyset is configured, so a SET pipelined before PostgreSQL
  confirms a mapping reset is not lost.
- Resets are queued even if the change-count lock is poisoned, so the
  batch's command tags stay in step.
…s main does (CIP-5021)

A SET CIPHERSTASH.KEYSET_ID that is not a UUID, sent while mapping is
disabled and no default keyset is configured, now returns the same
error as with mapping enabled instead of being logged and forwarded.
…s (CIP-5021)

Proxy applied SET CIPHERSTASH.KEYSET_ID, KEYSET_NAME and
UNSAFE_DISABLE_MAPPING when the statement was parsed, as well as each
time it ran. A client that prepared such a SET without executing it
changed Proxy's setting while PostgreSQL's stayed the same, so writes
could be encrypted under a keyset the client never selected.

Apply the setting only when a portal is bound from the statement.
…P-5021)

When a statement fails type checking and Proxy forwards the whole
Query, the SET CIPHERSTASH statements after it are now applied, in
order with their resets, as PostgreSQL runs them.

Also record in a test and the CHANGELOG that an invalid KEYSET_ID is
now refused while mapping is disabled, and that a prepared SET takes
effect only when executed.
Applying a prepared SET CIPHERSTASH setting only when it runs moved
the refusal of a keyset SET, with a default keyset configured or an
invalid KEYSET_ID, from Parse to Bind. The set_keyset_error
integration tests expect it at Parse.

Check the SET at Parse without applying it, so it is refused as
before.

tobyhede commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Review feedback audit at 3f746d6

I checked each review finding against the head commit by reading the code. CI is green on 3f746d6 (PostgreSQL 14 to 17).

@freshtonic's review

1. Fail-open: a reset was lost when mapping was disabled and the earlier batch's ReadyForQuery came between Bind and Execute. Fixed in c04f5ec, using the suggested fix of pruning by generation.

  • portal_resets and portal_deallocations now store the generation each entry was bound in.
  • At ReadyForQuery I, discard_portals_before prunes them by generation instead of by whether the portal is in portals (context/mod.rs:935-951). A portal bound for the next batch now keeps its effects even when the Bind adds no portal.
  • Completing an execution removes only effects bound at or before its generation (close_portal_bound_by, context/mod.rs:785-806).
  • Your scenario is now a regression test: a_reset_bound_before_the_previous_batch_is_ready_still_runs (tests.rs:669). The same scenario with mapping enabled is a_reset_bound_while_a_query_runs_still_runs_with_mapping_enabled (tests.rs:690).

2. In a simple Query, SET … TO DEFAULT undid a SET that came after it. Fixed in c04f5ec.

  • Each reset is queued with the change counts at its own position in the loop (push_settings_resets, frontend.rs:466 and :1011).
  • Tests cover both orders: a_set_after_set_to_default_in_one_query_is_kept (tests.rs:709) and set_to_default_after_a_set_in_one_query_resets_it (tests.rs:726), which is the mapping fail-open order.
  • The same fix for main is in test(proxy): cover settings resets and SET ordering cases (CIP-5021) #480.

3. A keyset SET sent before a mapping reset was confirmed was ignored. Fixed in c04f5ec and 6219c51.

  • While mapping is disabled, record_keyset_while_mapping_disabled (frontend.rs:1024-1031) applies keyset SETs, unless a default_keyset_id is configured, in which case a keyset can't be chosen anyway. An invalid KEYSET_ID is refused as it is with mapping enabled.
  • It is used in the Query loop, in the fallback after a statement fails type checking, and in the prepared path.
  • Test: a_keyset_set_sent_before_mapping_is_turned_back_on_is_kept (tests.rs:743).

4. Nit: the poisoned mutex. Fixed in c04f5ec. add_settings_resets_at and settings_change_counts recover from poisoning with PoisonError::into_inner and still queue the entries (context/mod.rs:1056, :1068-1072).

Out of scope: a simple Query with an encrypted statement drops its SET and DISCARD statements. This is tracked in CIP-5059.

Follow-up commits

  • 4a2f3d5: a prepared SET CIPHERSTASH.* is applied only when it runs, not at Parse. A prepared SET that is never executed no longer changes Proxy's settings.
  • 6fdc7ab: the SETs after a statement that fails type checking are still applied, in order with their resets, when the Query is forwarded unchanged.
  • 3f746d6: a keyset SET that would be refused is still refused at Parse, as the set_keyset_error integration tests expect, without being applied.

Codex


Generated by Claude Code

@tobyhede
tobyhede merged commit de2986e into release/2.2 Oct 21, 2026
4 checks passed
@tobyhede
tobyhede deleted the claude/project-thread-gvoja6-2.2 branch October 21, 2026 03:47
tobyhede added a commit that referenced this pull request Oct 22, 2026
…tions (CIP-5073)

Backport the transactional settings behaviour of #469 (CIP-4167) to
release/2.2, for CIP-5058. Proxy kept CIPHERSTASH.KEYSET_ID,
KEYSET_NAME and UNSAFE_DISABLE_MAPPING outside PostgreSQL's
transactions, so a rolled-back reset left Proxy with no keyset (and
later writes under default_keyset_id), a rolled-back SET of
UNSAFE_DISABLE_MAPPING = true left mapping disabled, and SET LOCAL
was treated as a plain SET.

The settings now follow PostgreSQL's rules:

- context/settings.rs is ported from main. ConnectionSettings keeps
  session and local values with transaction and savepoint snapshots:
  COMMIT keeps changes, ROLLBACK and ROLLBACK TO SAVEPOINT undo them,
  RELEASE keeps them, SET LOCAL ends with the transaction, and a failed
  transaction, explicit or implicit, is restored when it ends.
- Every settings command, including BEGIN, COMMIT, ROLLBACK and
  SAVEPOINT, is applied only when PostgreSQL answers it. Each statement
  of a Query that contains one, and every Execute, is queued by batch
  generation and matched to its own CommandComplete, EmptyQueryResponse
  or PortalSuspended, in order. A COMMIT answered with ROLLBACK rolls
  back, and an ErrorResponse skips the rest of the batch and fails the
  transaction. This replaces the per-setting change counters and reset
  queue added in #475, and applying a prepared SET at Bind.
- The frontend maps each statement with the settings projected over the
  commands PostgreSQL has not answered yet, including the statements
  before it in the same Query, and encrypts with that keyset. The
  backend decrypts with the confirmed keyset.
- Setting names are matched as PostgreSQL matches them,
  UNSAFE_DISABLE_MAPPING accepts every PostgreSQL boolean spelling,
  and an unknown CIPHERSTASH setting or a value Proxy cannot interpret
  is refused, as on main. RESET of either keyset setting clears the
  keyset, as on main.
- A simple query that Proxy rewrites now forwards every statement.
  Statements Proxy did not map, such as a SET, were dropped from a
  rewritten query, which would also have put the responses out of step
  with the queued settings commands.

The pooler test matrix, the settings protocol journal and the
pipelining barriers of #469 are specific to main's architecture and
are not ported.

Tests: the settings.rs unit tests from #469, transaction tests for the
frontend and backend together, and #478's rolled_back_mapping_changes_
are_undone and multitenant rolled_back_keyset_changes_are_undone
integration tests. The #475 tests still pass; the two that expected a
reset to survive a failed implicit transaction now expect it to roll
back, as on main.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants