feat: add IBigSegmentStore interface + Redis and DynamoDB stores#536
feat: add IBigSegmentStore interface + Redis and DynamoDB stores#536beekld wants to merge 10 commits into
Conversation
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0082f6e. Configure here.
| return std::unique_ptr<RedisBigSegmentStore>(new RedisBigSegmentStore( | ||
| std::make_unique<sw::redis::Redis>(std::move(uri)), | ||
| std::move(prefix))); | ||
| } catch (sw::redis::Error const& e) { |
There was a problem hiding this comment.
Should we catch std::exception as well?
| public: | ||
| DynamoDBBigSegmentTests() | ||
| : table_name_("ld-dynamodb-big-segments-test"), | ||
| prefix_("testprefix"), |
There was a problem hiding this comment.
Should we have a test for an empty prefix?
| std::make_unique<sw::redis::Redis>(std::move(uri)), | ||
| std::move(prefix))); | ||
| } catch (sw::redis::Error const& e) { | ||
| return tl::make_unexpected(e.what()); |
There was a problem hiding this comment.
Do we need to sanitize this for the redis user/password?
There was a problem hiding this comment.
Or maybe make this and the dynamo version into something vague? Main concern is just a URI coming out which includes user/password or something else the customer puts into the URI and considers secret.
There was a problem hiding this comment.
I wanna push back on this one. I think making it vague would lose the diagnostic value of the original error messages. I would hope that redis already wouldn't be putting sensitive info into error messages. But also, we aren't logging this here, either. I think the "correct" solution would be to have a sanitizer wherever we are logging, but that's a much bigger task.
| IBigSegmentStore::GetMembershipResult RedisBigSegmentStore::GetMembership( | ||
| std::string const& context_hash) const { | ||
| std::string const include_key = | ||
| prefix_ + ":" + kIncludeKeyNamespace + ":" + context_hash; |
There was a problem hiding this comment.
nit: Would it be work it to cache prefix_ + ":" + kIncludeKeyNamespace and prefix_ + ":" + kExcludeKeyNamespace then just add on the context_hash?
| std::string prefix) | ||
| : redis_(std::move(redis)), | ||
| prefix_(std::move(prefix)), | ||
| sync_time_key_(prefix_ + ":" + kSyncTimeKey) {} |
There was a problem hiding this comment.
Should we check for an empty prefix?
.Net and JS do, but Java doesn't.
public RedisStoreBuilder<T> Prefix(string prefix)
{
_prefix = string.IsNullOrEmpty(prefix) ? Redis.DefaultPrefix : prefix;
return this;
}There was a problem hiding this comment.
Done
Yes, I think Java is probably wrong here
|
|
||
| auto const it = item.find(kBigSegmentsSyncTimeAttribute); | ||
| if (it == item.end()) { | ||
| return tl::make_unexpected( |
There was a problem hiding this comment.
Other implementations seem more lenient here. Though I wonder if there was an evolution where maybe this field didn't exist originally.
There was a problem hiding this comment.
The spec seems to say we should be more lenient too. Done.

Ticket: SDK-2363 · Follows #534
Summary
Adds the public
IBigSegmentStoreinterface, theMembership/StoreMetadatavalue types, and concrete DynamoDB + Redis implementations. Schema strings match what the Relay Proxy writes.Design notes
synchronizedOnparsing. Stored as DynamoDB N / Redis string. Both stores reject malformed values (non-numeric strings, wrong DynamoDB attribute type) rather than silently returning 0, matching the existingdynamodb_source.cpprow validation.Not in scope
BigSegmentsBuilderconfig plumbing.BigSegmentStoreWrapper(LRU cache + staleness polling) and the hashing path.rules.cppbig-segments TODO.Test plan
Membershipunit tests pass.RedisBigSegmentStoreintegration tests pass against Redis 7 (docker).DynamoDBBigSegmentStoreintegration tests pass against DynamoDB Local.Note
Medium Risk
New evaluation-adjacent persistence API with strict error handling on malformed store data; behavior is well-tested but not yet connected to flag evaluation, so misconfiguration would surface only after later wiring.
Overview
Introduces a Big Segments read path for the server SDK: public
IBigSegmentStoreplusMembership/StoreMetadata, with Redis and DynamoDB stores that match Relay’s key/schema layout (prefix-scoped, read-only).GetMembershiploads included/excluded segment refs per hashed context;GetMetadatareads the sync timestamp for staleness. DynamoDB uses consistentGetItem, validates String Set / Number attribute types, and errors on malformed rows instead of silent empty data; Redis uses set membership and string sync time with the same strict parsing.Membership::FromSegmentRefsapplies spec rules (inclusion over exclusion). Integration tests cover empty stores, prefixes, and bad data; config, caching wrapper, hashing, and evaluator wiring are not in this PR.Reviewed by Cursor Bugbot for commit b0ab11c. Bugbot is set up for automated code reviews on this repo. Configure here.