feat: expose refresh parameter in ElasticsearchDocumentStore and OpenSearchDocumentStore #2613
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
github-actions
bot
added
integration:elasticsearch
integration:opensearch
type:documentation
…SearchDocumentStore Add configurable refresh parameter to write, delete, and update methods in both ElasticsearchDocumentStore and OpenSearchDocumentStore. This allows users to control when index changes become visible to search operations, enabling read-your-writes consistency without relying on time.sleep() workarounds. Methods updated: - write_documents / write_documents_async - delete_documents / delete_documents_async - delete_all_documents / delete_all_documents_async - delete_by_filter / delete_by_filter_async - update_by_filter / update_by_filter_async The refresh parameter accepts: - True: Force immediate refresh - False: No refresh (best for bulk performance) - "wait_for": Wait for next refresh cycle (default) Closes deepset-ai#2065
GunaPalanivel
force-pushed
the
feature/elasticsearch-2605-expose-refresh-param
branch
from
94569c0 to
9a62b54
Compare
Member
|
@GunaPalanivel thanks for the contribution. Please split your work into two distinct PRs, one targeting Elasticsearch and the other one about OpenSearch. This would help with the release process of this monorepo. |
Contributor
Author
|
Thanks for the suggestion @anakin87! Done - I've split this into two separate PRs: |
Member
|
Ok, thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues
Closes #2065
Description
This PR exposes the
refreshparameter to relevant methods inElasticsearchDocumentStoreandOpenSearchDocumentStore, allowing users to control when index changes become visible to search operations.Motivation
Both Elasticsearch and OpenSearch are "near real-time" search engines where changes are not immediately visible after write/update/delete operations. Previously, users had to use
time.sleep()workarounds to ensure consistency. This change exposes the underlyingrefreshparameter, giving users explicit control over this behavior.Changes
New
RefreshTypeparameter added to the following methods in both document stores:write_documents/write_documents_async"wait_for"delete_documents/delete_documents_async"wait_for"delete_all_documents/delete_all_documents_asyncTruedelete_by_filter/delete_by_filter_async"wait_for"update_by_filter/update_by_filter_async"wait_for"Parameter values:
True: Force refresh immediately after the operationFalse: Do not refresh (better performance for bulk operations)"wait_for": Wait for the next refresh cycle (default, ensures read-your-writes consistency)Test Updates
Updated integration tests to use
refresh=Trueinstead oftime.sleep(), making tests more reliable and faster.Additional Fixes
delete_all_documents_asyncin OpenSearch to setwait_for_completion=True(matching Elasticsearch behavior), ensuring the async delete operation completes before returning# ruff: noqa: FBT001, FBT002comment to OpenSearch to match Elasticsearch's linting configurationasyncioimport from Elasticsearch testsHow was this tested?
refreshparameterChecklist