feat: expose refresh parameter in OpenSearchDocumentStore #2623
+107
−42
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add configurable refresh parameter to write, delete, and update methods in OpenSearchDocumentStore. This allows users to control when index changes become visible to search operations, enabling read-your-writes consistency without relying on time.sleep() workarounds. Methods updated: - write_documents / write_documents_async - delete_documents / delete_documents_async - delete_all_documents / delete_all_documents_async - delete_by_filter / delete_by_filter_async - update_by_filter / update_by_filter_async The refresh parameter accepts: - True: Force immediate refresh - False: No refresh (best for bulk performance) - 'wait_for': Wait for next refresh cycle (default) Additional fix: - Fixed delete_all_documents_async to set wait_for_completion=True, ensuring the async delete operation completes before returning Closes deepset-ai#2065
github-actions
bot
added
integration:opensearch
type:documentation
Member
|
Thanks for opening a separate PR. Let's put this on hold and first agree on #2622 |
Closed
3 tasks
Member
|
@GunaPalanivel when you have time, feel free to adapt this PR based on #2622 🙂 |
Contributor
Author
|
I'm working on it 🙂 |
…ings to match ES pattern (deepset-ai#2623)
Contributor
Author
|
Based on #2622 , I've adapted this PR for OpenSearch: Main changes:
Let me know if I missed anything or got something wrong, happy to fix! 🙂 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues
Part of #2605
Description
This PR exposes the
refreshparameter to relevant methods inOpenSearchDocumentStore, allowing users to control when index changes become visible to search operations.Motivation
OpenSearch is a "near real-time" search engine where changes are not immediately visible after write/update/delete operations. Previously, users had to use
time.sleep()workarounds to ensure consistency. This change exposes the underlyingrefreshparameter, giving users explicit control over this behavior.Changes
New
RefreshTypeparameter added to the following methods:write_documents/write_documents_async"wait_for"delete_documents/delete_documents_async"wait_for"delete_all_documents/delete_all_documents_asyncTruedelete_by_filter/delete_by_filter_async"wait_for"update_by_filter/update_by_filter_async"wait_for"Parameter values:
True: Force refresh immediately after the operationFalse: Do not refresh (better performance for bulk operations)"wait_for": Wait for the next refresh cycle (default, ensures read-your-writes consistency)Test Updates
Updated integration tests to use
refresh=Trueinstead oftime.sleep(), making tests more reliable and faster.Additional Fixes
delete_all_documents_asyncto setwait_for_completion=True, ensuring the async delete operation completes before returningHow was this tested?
refreshparameter