Stream based OnDiskGraphIndexCompactor API flavor #722

Introduction

This change #659 introduced a new OnDiskGraphIndexCompactor into jvector library that uses disk based graph (instead of heap based) for compaction (merge). In opensearch-jvector plugin, we would heavily benefit from this feature (see please opensearch-project/opensearch-jvector#649) however the APIs which OnDiskGraphIndexCompactor have some limitations.

OnDiskGraphIndexCompactor API

At a high level, OnDiskGraphIndexCompactor (and internal CompactWriter) assumes that final output has to be a file (it is represented as a Path but used later on to create FileChannels and MappedByteBuffers. In contrary, Apache Lucene (that opensearch-jvector plugin uses under the hood) abstracts the I/O over IndexInput / IndexOuput (could be though of as in/out streams). Moving off file / file channels towards streaming APIs would allow OnDiskGraphIndexCompactor to be integrated into opensearch-jvector plugin (and very likely, simplify a lot Cassandra integration).

Implementation

Just replacing Path with IndexInput / InputStream / RandomAccessWriter or alike will not be enough (primarily due FusedPQ path that relies on FileChannel / MappedByteBuffer). Since opensearch-jvector plugin does not implement FusedPQ at the moment, the idea is to develop the prototype that uses steaming APIs without FusedPQ support. This is reasonable simplification for the initial POC in order to focus on feasibility of the OnDiskGraphIndexCompactor in general. Once and if proven possible, the end goal is full feature parity.

The rough API changes sketch (FusedFeature is out, Path -> IndexWriter):

CompactWriter(IndexWriter writer,
                  int maxOrdinal,
                  int numBaseLayerNodes,
                  long startOffset,
                  List<CommonHeader.LayerInfo> layerInfo,
                  int entryNode,
                  int dimension,
                  List<Integer> layerDegrees)

Risks

The major risks that the API changes suggested in the implementation section may bring is performance degradation, particularly on the operating system(s) where FileChannel are implemented efficiently. The performance impact to be measured and, at worth, may lead to two different flavours (file / file channels and streams).

The hard constraint on Apache Lucene side - no random I/O on the write path (see please apache/lucene#15420), the CompactWriter does heavily rely on random access at the moment (through RandomAccessWriter).

Benefits

Why OnDiskGraphIndexCompactor is worth integrating into opensearch-jvector plugin? There are few reasons:

• merge large graphs without OOMing the JVM
• reuse the jvector implementation instead of supporting our own (leading segment merge)