fix: contain Text equality and numeric filter values in their own clause - #721
Conversation
There was a problem hiding this comment.
LGTM. The "replace rather than escape" reasoning for " and coercing instead of trusting isinstance are both right, and the integration test on != is the only thing that could catch the unmatchable-phrase case.
Just one thing to point out, not a blocker for this PR: Geo has the same containment gap and isn't covered here. GeoSpec.__init__ validates only unit, so longitude and latitude reach Geo.__str__'s %s slots unvalidated, and a string coordinate can close its clause and inject a |.
|
Opened #723 for the |
`Text.__str__` interpolated the caller's value straight into its operator
templates, so a value carrying a `"` could terminate the quoted phrase it was
meant to sit inside and append arbitrary RediSearch syntax. Under DIALECT 2 an
injected `|` binds looser than the implicit space-AND, so it lifts to the root
of the parse tree and any surrounding filter stops constraining the query.
`Num` had the same defect by a different route: `_set_value` type-checks
without coercing, so a numeric subclass overriding `__str__` satisfies the
check and injects when formatted, and `between` never reached that check at all.
Text equality and inequality now replace `"` with a space. Escaping it does not
work, because escaping is symmetric: a backslash joins the separator into the
term, so `@f:("say \"hi\" now")` asks for a term containing a quote and RedisVL
writes documents unescaped. On `==` that matches nothing; on `!=` the
unmatchable phrase makes the negation match everything, so an exclusion filter
silently stops excluding. A value of nothing but quotes now renders `*`, for
the same reason. Only the quote is replaced: a trailing backslash does not
escape the closing quote — `@f:("x\")` parses as the term `x\` — and replacing
it would break matching against documents that were written escaped.
`%` still interpolates raw, which is what makes `*`, `%%` and `|` work, and its
docstring now says so. `Text` lists the one raw operator rather than deriving
the quoted ones from the templates, so a new operator, including one added by a
subclass, is contained unless it opts out.
All seven `Num` operators now coerce their value to a builtin `int` or `float`
after the type check, which is what defeats the hostile-`__str__` case;
`numbers.Real` keeps numpy scalars working. `NaN` is rejected, since it renders
a query RediSearch refuses. The unreachable `BETWEEN` `OPERATOR_MAP` entry, its
`__str__` branch, and the `tuple` in `SUPPORTED_VAL_TYPES` that only that branch
consumed are all deleted.
The MCP locked-filter backstop needed a matching change. It counted bracket
depth without knowing quotes exist, which was only safe while the filter
boundary escaped every paren and brace in a text value; without that it refused
ordinary values such as `smiley :)` and `a[b`. It now skips a quoted phrase
wholesale, ending at the first unescaped quote so a value that did reach the
rendering raw is still counted. Two near-identical scanners collapse into
`_find_unescaped`, which takes a bound — without one the range branch pays for
the whole remaining string on every `[`, which measured 14s of blocking CPU on
a 1 MB rendering.
The boundary escaping in `redisvl/mcp/filters.py` is removed for eq/ne/in,
which is required rather than cleanup: with `Text` also handling the value it
would be treated twice and mangled. That also fixes a live defect, since the
escaper escaped the space and so rendered any multi-word value as a literal
that matched nothing. `like` keeps its boundary escaping, because the library
leaves `%` raw.
b239d96 to
b9f43bd
Compare
`_reject_escapable_filter`'s quoted-phrase scan treated `\` as escaping the
next character, so a text `eq`/`ne` value ending in a backslash consumed the
template's own closing quote as an escape pair. The scan then found no
terminator, read the phrase as unterminated, and refused the filter. Any value
ending in an odd number of backslashes was affected, an ordinary Windows path
among them, on every tool with a locked filter.
RediSearch does not honour that escape either: `@f:("x\")` closes the phrase and
yields the term `x\`. The scan now matches the parser and stops at the next
quote regardless of what precedes it. Nothing is given up, because a value
cannot contribute a quote of its own -- `Text` replaces it, and `Tag` and `like`
escape theirs outside any phrase -- so every quote in a rendering the DSL can
produce is a template delimiter.
Verified against a live index: of 1816 renderings the scan accepts, drawn from an
alphabet of quotes, backslashes, parens, pipes and a tag clause, none returned a
document outside the locked tenant, and 176 failed closed as server-side syntax
errors.
Leaving the renderer alone is deliberate. Stripping a trailing backslash in
`Text` would have fixed the symptom and broken matching, because the two cases
differ: a value ending in one backslash matches documents that index the bare
term, while two backslashes index a literal backslash and need it kept.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3c21906. Configure here.
…ng filter `Text.__str__` returned `*` when a neutralized `==`/`!=` value held nothing but whitespace. That inverted the operator it was meant to protect: an empty phrase matches no document, so `==` should fail closed, and returning `*` made it match every one. `format_expression` also drops a `*` operand, so the clause vanished from any surrounding AND rather than narrowing it. The guard bought nothing on the branch it was written for. Measured on Redis 8.4.5 at DIALECT 2, `(-@v:(" "))` already matches every document, exactly as `*` does, so `!=` was unaffected either way. Substituting a space can never empty a non-empty value, so the phrase always holds a character and cannot reach the `INDEXEMPTY` error a literal `@field:("")` raises -- which is the only crash the guard could have averted.
Only tests/unit/test_filter.py conflicted, on the import list: main's #717 and #721 added FilterExpression and FilterOperator, this branch added render_filter and intersect_with_filter. Resolved as the union of both. The two sides are orthogonal. Main's work escapes filter *values* so they cannot break out of their own clause; this branch parenthesises the rendered filter so a union inside it cannot bind across an intersection. Verified after merging that render_filter leaves every newly-escaped rendering byte-identical, and that the wildcard contract it relies on still holds: str(FilterExpression("*")) is "*", and a bare un-narrowed FilterField still renders to None without being mutated.
…#724) Fixes #723 ### Summary `GeoSpec` validated only its `unit`. `longitude` and `latitude` were stored as given despite their `float` annotations, and every one of the four arguments is interpolated into a RediSearch query by `Geo.__str__`. A `str `coordinate carrying a `]` closed the geo clause and had its remainder parsed as syntax. Similar to #721, an injected `|` binds looser than the implicit space-AND, so it lifts to the root of the parse tree and any filter sharing the query stops constraining it: ``` crafted = "-122.4194 37.7749 10 km] | @secret:{leaked}" str(Geo("location") == GeoRadius(crafted, 37.7749, 1)) # before: @location:[-122.4194 37.7749 10 km] | @secret:{leaked} 37.7749 1 km] # after: TypeError: GeoRadius longitude must be an int, a float, or another # numbers.Real; got str ``` The `unit` had the same shape by a different route, and this one is not in the issue as filed. `unit.lower()` not in `self.GEO_UNITS` is an equality test the caller controls, while `self._unit = unit.lower()` stored the caller's object. A `str` subclass is neutralised by `str.lower()` returning a builtin, but an object whose `lower()` returns `self` and whose `__eq__` matches `"km"` passed the check and then rendered its own `__str__`: ``` class Kilometres: def lower(self): return self def __eq__(self, other): return other == "km" def __hash__(self): return hash("km") def __str__(self): return "km] | @secret:{leaked}" str(Geo("geo_field") == GeoRadius(1.0, 2.0, 3, Kilometres())) # before: @geo_field:[1.0 2.0 3 km] | @secret:{leaked}] # after: @geo_field:[1.0 2.0 3 km] ``` Coordinates were also unbounded, so GeoRadius(9999, -9999, 1) and a NaN coordinate both rendered queries the server rejects, failing far from the line that caused them. ### Changes 1. Coordinates are coerced and range-checked at the constructor `GeoSpec.__init__` routes both coordinates through `_coerce_to_number_within`, which coerces to a builtin and requires a finite value inside an inclusive range — `LONGITUDE_RANGE = (-180.0, 180.0)` and `LATITUDE_RANGE = (-90.0, 90.0)`, class attributes beside the existing `GEO_UNITS` so the numeric and unit domains share a namespace and a subclass can widen either. Constructor-time rather than render-time is deliberate. The error names the offending argument at the caller's line instead of surfacing from inside `__str__`, and `str()` is on the query path so it should not re-validate on every call. The bounds are inclusive because the antimeridian and the poles are real places. `isfinite` is checked separately rather than left to the range comparison, so an infinite bound could not admit an infinite value. That much is unreachable through this class's own finite ranges and is covered by a subclass test. The ordering is load-bearing for a second and entirely reachable reason, though: `isfinite` raises `OverflowError` on an int too large to convert to a float, so the range is compared first and rejects `10**400` as the documented `ValueError`. 2. The unit stores `GEO_UNITS' own spelling `_canonical_unit` returns the matched element of `GEO_UNITS` rather than the value that matched it. The caller's `__eq__` still decides whether a unit matches; it no longer supplies what renders. There is deliberately no `isinstance(unit, str)` check, which would reject a legitimate `str` subclass — canonicalisation makes the identity of `lower()`'s return value irrelevant, since it is only ever compared, never rendered. `GEO_UNITS `stays the single source of truth for both the lookup and the error message. Unit validation still runs before the coordinates, so a caller passing both a bad unit and a bad coordinate sees the error they saw before. 3. The radius is guarded explicitly, and stops being truncated `%i` was doing two jobs badly. As a type guard it worked by accident, raising `TypeError` on a `str`, which left a format specifier load-bearing where nothing documented it. As a formatter it was simply wrong: it truncates toward zero, so a fractional radius silently queried a smaller circle than the caller asked for, and a sub-unit radius rendered `0`, which the server rejects outright. Measured on 8.4.5 against a document 1.5 km from the centre: | Caller writes | Rendered before | Hits | Rendered now | Hits | |---|---|---|---|---| | `radius=1.9, unit="km"` | `1 km` | 1 | `1.9 km` | 2 | | `radius=0.5, unit="km"` | `0 km` | `Invalid GeoFilter radius` | `0.5 km` | 1 | So the radius now renders through `%s`, and `GeoRadius` coerces it — that coercion is the type guard, explicitly, with a test that fails without it. An integral float renders as an int, because `repr` switches to exponent form at 1e16 and `1e+16` is a syntax error at DIALECT 1, the Redis 8 server default. The radius is also bounded now: zero, negative and infinite are refused at the constructor. Measured, the server answers `Invalid GeoFilter radius` to zero and to a negative, and an infinite radius previously escaped as an `OverflowError` from inside `Geo.__str__`. The bound is a chained comparison rather than `math.isfinite`, which raises `OverflowError` on an int too large to convert — a radius has no upper bound to reject such a value first, unlike a coordinate. 4. The numeric coercion moves to module scope `Num._coerce_numeric` from #721 becomes the module-level `_coerce_to_number`, because `GeoSpec` is not a `FilterField` and could not inherit it. `Num._coerce_numeric` stays as a one-line delegator binding `cls.__name__`, so error text is unchanged and `Timestamp` still reports its own name through `cls`. ### Tests 30 geo cases in `tests/unit/test_filter.py`, sharing a `_geo_radius(**overrides)` builder so each row names only the argument it changes. `_StrOverridingInt`/`_StrOverridingFloat` are reused from #721 rather than duplicated per filter type. Every guard was mutation-checked by reverting it alone, including both `OPERATOR_MAP` templates separately, since `__ne__` is otherwise asserted nowhere. The geo filter in `tests/integration/test_query.py` now uses a fractional radius, so a truncation regression fails against a real server rather than only against a string comparison. Verified: reverting the template alone turns that test red with `Invalid GeoFilter radius`. ## Release Notes Geo filter arguments are now validated where they are supplied, rather than reaching the query string unchecked. A `str` coordinate could previously close the geo clause and have its remainder parsed as RediSearch syntax, so a filter built from untrusted input could be widened past the scope it was meant to enforce. Upgrade if any geo filter argument in your application originates from user input. One change moves results silently and deserves attention before you upgrade. A fractional radius used to be truncated toward zero, so `GeoRadius(lon, lat, 1.9, "km")` queried a 1 km circle; it now queries 1.9 km and returns the rows it always should have. Nobody gets an error from this, only different rows. Three changes are backwards-incompatible, all of them turning a value that used to build a broken query into an error at the line that supplied it. `Decimal` coordinates now raise `TypeError`, having rendered successfully before, so a coordinate arriving from a `NUMERIC` database column or from `json.loads(payload, parse_float=Decimal)` has to be passed through `float()` first. A zero, negative or infinite radius now raises `ValueError`, where zero and negative previously built a query the server rejected with `Invalid GeoFilter radius` and infinite raised `OverflowError` from inside the formatter. And a coordinate outside its range now raises `ValueError` instead of building a query the server refuses. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Security-sensitive query construction changes plus behavior shifts: fractional radii affect results, and some previously accepted values (`Decimal`, out-of-range coords, zero/negative radius) now raise at construction time. > > **Overview** > **Geo filters no longer accept untrusted values that could break out of the geo clause** (e.g. injected `]` or `|` widening the query). `GeoSpec` / `GeoRadius` now coerce longitude, latitude, and radius to builtin numbers, enforce coordinate ranges and a positive finite radius, and store a canonical unit from `GEO_UNITS` instead of the caller’s object. > > **Radius rendering is fixed:** `Geo` templates use `%s` for radius instead of `%i`, so fractional radii are no longer truncated (including sub-unit values that used to render as `0` and fail on the server). Shared module helpers `_coerce_to_number` / `_coerce_to_number_within` back geo validation; `Num._coerce_numeric` delegates to them. > > **Tests** add broad unit coverage for coercion, injection, and edge cases; integration geo filters use a `0.5 km` radius to catch truncation against Redis. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 6b53a4b. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Vishal Bala <vishalbala.1994@gmail.com>
|
🚀 PR was released in |

Motivation
A filter value could terminate the clause it was rendered into and append arbitrary RediSearch syntax.
Text.__str__interpolated the caller's value straight into its operator templates, so a value carrying a"closed the quoted phrase early; under DIALECT 2 an injected|binds looser than the implicit space-AND, so it lifts to the root of the parse tree and any surrounding filter stops constraining the query. Measured on Redis 8.4.5 against an index holding two tenants,str((Tag("tenant") == "acme") & (Text("v") == crafted))returned the other tenant's document andFT.EXPLAINCLIreported a top-levelUNION.Numhad the same class of defect by a different route:_set_valuetype-checked without coercing, so a numeric subclass overriding__str__satisfied the check and injected when formatted, andbetweennever reached that check at all.The library is the exposed surface. The MCP server escaped text at its own boundary and was never affected, but any caller building a filter from untrusted input was.
Changes
Text equality replaces the quote rather than escaping it
Escaping does not work here, and the reason decides the whole design. RediSearch tokenization is symmetric: punctuation separates tokens on both sides of the wire, and a backslash joins the separator into the term. Since RedisVL writes documents unescaped,
@f:("say \"hi\" now")asks for a term containing a quote that no document ever stored. On==that matches nothing. On!=the unmatchable phrase makes the negation match everything, so an exclusion filter silently stops excluding, which is strictly worse than the injection it would have prevented.Equality and inequality therefore replace
"with a space, which is what the tokenizer left at that position anyway. A value of nothing but quotes renders*, for the same reason. Only the quote is replaced: a trailing backslash does not terminate a phrase either, since@f:("x\")parses as the termx\, and replacing it would break matching against documents that were written escaped.%continues to interpolate its value raw, which is what makes*,%%and|work, and its docstring now says so and points a caller holding untrusted input at==.Textlists that one raw operator rather than deriving the quoted ones from the templates, so a new operator is contained unless it opts out. That covers one added by a subclass, which a set of quoted operators computed in the base class body would miss.Numeric values are coerced, not merely type-checked
All seven
Numoperators now pass their value through_coerce_numericafter the type check. Coercion is the guard rather than the check: every numeric value is formatted into the query string, soint()andfloat()returning builtins is what strips a subclass's__str__override.numbers.Realkeeps numpy scalars working, which a concrete(int, float)check would have rejected. NaN is refused, since@field:[nan ...]is a query RediSearch rejects outright.The unreachable
BETWEENentry inOPERATOR_MAP, its branch inNum.__str__, and thetupleinSUPPORTED_VAL_TYPESthat only that branch consumed are deleted. Nothing assignedFilterOperator.BETWEENto an instance, which is whybetweenlooked like it was bypassing machinery that worked.The MCP locked-filter backstop learns about quoted phrases
_reject_escapable_filtercounted bracket depth without knowing quotes exist. That was safe only while the filter boundary escaped every parenthesis and brace in a text value; without it the guard refused ordinary values such assmiley :)anda[b. It now skips a quoted phrase wholesale, because inside quotes a parenthesis is literal text and a|is a separator rather than a union. Ending the skip at the first unescaped quote is what keeps it failing closed: a value that did reach the rendering raw ends the phrase there, and the injected remainder is counted as before.Two near-identical scanners collapse into
_find_unescaped, which takes a bound. Without one the range branch pays for the whole remaining string on every[, which made the walk quadratic in the number of spans. Measured at 14 seconds of blocking CPU on a 1 MB rendering, that is enough to stall the server's event loop for every concurrent request.The MCP text boundary stops escaping eq, ne and in
Required rather than cleanup: with
Textalso handling the value it would be treated twice and mangled. This also fixes a live defect, because the boundary escaper escaped the space and so rendered any multi-word value as a literal that matched nothing.likekeeps its boundary escaping, since the library leaves%raw.docs/concepts/mcp.mdno longer attributes text escaping to the filter boundary, and no longer names a parenthesis as able to close a clause.==and!=match a literal phrase while%takes a raw pattern.Tests
Four test functions in
tests/unit/test_filter.pywere defined twice, so the earlier definition of each never collected.Num.betweenwith itsinclusive=variants andTag != <falsy>had coverage only there and are ported forward; two of the dead assertions were simply wrong, which is independent evidence they had never run.Because the existing suite passed identically with the fix applied and reverted, each guard is mutation-checked by reverting it alone. Fourteen mutations are each caught by a named test. The integration suite gains one row asserting that a quote-bearing value on
!=excludes its document, which is the one claim no rendering assertion can make: escaping the quote instead returns the whole corpus.Notes
Two behaviour changes reach existing callers. A
Text==or!=value containing a"now renders differently, and a value of only quotes selects everything instead of producing a malformed query.Num.betweennow raisesTypeErroron astrorDecimalendpoint, both of which rendered successfully before, since neither is anumbers.Real; theRaisesdocstring names them explicitly. No working caller regresses on any other input.The one change that widens a result set is the MCP boundary fix, and it stays inside the lock: a structured text
eq,neorinfilter whose value contains a space went from matching nothing to matching correctly. Operators running a profile with a locked multi-word text filter should expect that scope to start applying.Timestamp.betweenoverridesNum.betweenand calls the formatter directly, so it does not inherit the endpoint coercion. It is not injectable, because_convert_to_timestampraises on a non-ISO string, butbetween(None, None)renders@ts:[None None]and NaN renders@ts:[nan ...], both server-side syntax errors. Deliberately out of scope and filed separately.Tag value handling is deliberately untouched here; #717 addressed that separately and has merged, so this branch now sits on top of it.
Geohas the same missing-validation shape asNumdid, sinceGeoSpec.__init__validates onlyunitand leaves the coordinates to reachGeo.__str__unchecked. Tracked in #723 rather than folded in.Three changes to the filter layer are in flight together: this one, #717 for tag values, and #720 for filter clauses in vector query strings. All three are independent, all target
main, and none needs to merge before another on functional grounds. #717 merges cleanly against both of the others. This branch and #720 both touchredisvl/query/filter.pyandtests/unit/test_filter.py, in different regions.git merge-treereports a single conflict, in thetests/unit/test_filter.pyimport block, which both changes rewrote; whichever merges second resolves it in one hunk. The two fixes are complementary and neither substitutes for the other: #720 stops a legitimately built union binding across an intersection at assembly time, and verified against a live server, its parenthesisation does not contain the injection this change fixes: as a KNN pre-filter the crafted value still explains as a rootUNION.Next Steps
tests/unit/test_filter.pyimport conflict against fix: parenthesise filter clauses in vector query strings #720, whichever of the two lands second.Timestamp.betweenendpoint validation, andTokenEscaper.escaped_chars_no_wildcard_recompiling from the class constant and so ignoring an injected pattern.Release Notes
Query filter values are now contained in the clause they render into. A value containing a double quote could previously close its own clause and have the remainder parsed as RediSearch syntax, so a filter built from untrusted input could be widened past the scope it was meant to enforce. Upgrade if any filter value in your application originates from user input.
One backwards-incompatible change comes with it.
Num(field).between(start, end)now raisesTypeErrorunless both bounds are real numbers. AstrorDecimalbound rendered successfully before and now fails, so a caller passing a numeric string from a request body, or aDecimalfrom a database column, has to coerce it first.int,floatand numpy scalars are unaffected.One fix moves result counts rather than breaking anything.
Text(field) != valuewith a quote-bearing value previously matched every document, so the filter excluded nothing at all; it now excludes correctly.For the MCP server, structured text filters carrying a multi-word value were escaped in a way that matched nothing.
eq,neandinnow work as intended, andlikeis unchanged.Text(field) % patternis deliberately unchanged and still interpolates its value raw, so do not pass untrusted input to it.