[CALCITE-7755] Support IEJoin for inequality joins by zzwqqq · Pull Request #5233 · apache/calcite

zzwqqq · 2026-08-30T09:30:20Z

Jira Link

Changes Proposed

IEJoin is an algorithm for joins with two inequality predicates, described in “Lightning Fast and Space Efficient Inequality Joins,” PVLDB 8(13), 2015.

This change adds an IEJoin implementation for the Enumerable convention. It applies to inner joins with at least two inequality predicates between fields from the two inputs. IEJoin handles the first two predicates; any remaining predicates are evaluated by an EnumerableCalc. Other join conditions are left to the existing rules.

Planner and execution tests are included.

mihaibudiu · 2026-09-05T17:09:20Z

+ */
+package org.apache.calcite.linq4j;
+
+/** Comparison operator applied to the left and right keys of an inequality


I think this class is fully generic, maybe it can find uses outside of joins. So it's enough to characterize it as "comparison operator".

That being said, is there a benefit reusing the kind?

Thanks! I wanted to use SqlKind at first, but that would create a circular dependency. I just realized we already have ExpressionType.

mihaibudiu · 2026-09-06T01:24:37Z

+        (leftPoint, rightPoint) -> leftPoint.name + ":" + rightPoint.name);
+  }
+
+  private static List<String> nestedLoop(List<Point> leftRows,


I assume this computes the expected values. A little JavaDoc would help

Yes, that’s right. Added Javadoc, thanks.

mihaibudiu · 2026-09-06T01:24:52Z

+        actual, is(expected));
+  }
+
+  private static Enumerable<String> ieJoin(List<Point> left, List<Point> right,


I assume this computes the result using the new enumerator.
Add JavaDoc

I assume this computes the result using the new enumerator. Add JavaDoc

Yes. Added Javadoc, thanks.

mihaibudiu · 2026-09-06T01:27:05Z

+  private @Nullable TResult current;
+  private boolean hasCurrent;
+
+  IEJoinEnumerator(Enumerable<TLeft> left, Enumerable<TRight> right,


This needs some JavaDoc.

Thanks, added Javadoc.

mihaibudiu · 2026-09-06T01:27:31Z

+    implements Enumerator<TResult> {
+  private final List<TLeft> leftRows = new ArrayList<>();
+  private final List<TRight> rightRows = new ArrayList<>();
+  private final List<Entry<TKey1, TKey2>> firstOrder;


some of these fields are not obvious, can you please document them?

some of these fields are not obvious, can you please document them?

Thanks, I've added comments for these fields.

mihaibudiu · 2026-09-06T01:28:30Z

I don't think you can assume that people who review this code have read the papers; try to add enough information to make the code self-explanatory as much as possible.

zzwqqq · 2026-09-06T10:45:17Z

I don't think you can assume that people who review this code have read the papers; try to add enough information to make the code self-explanatory as much as possible.

Thanks for reviewing! I've added a short explanation and an example to walk through the algorithm.

mihaibudiu

Why doesn't this work for other types, like DOUBLE?

mihaibudiu · 2026-09-09T03:33:41Z

+
+  /** Creates an EnumerableIEJoin.
+   *
+   * <p>Use {@link #create} unless you know what you're doing. */


Is this comment still relevant, or is it just a copy-paste from other implementations?
Usually you would write this on a constructor if the other method creates some other important invariants.

Thanks. I've removed the comment.

mihaibudiu · 2026-09-09T03:35:00Z

+      outputRows = RelMdUtil.addEpsilon(outputRows);
+    }
+    final double inputRows = leftRows + rightRows;
+    // IEJoin sorts the union twice, scans it once, then emits the result.


why would you sort something twice?

For l.x < r.x AND l.y > r.y, one sort is by x and the other by y. I've updated the comment to make that clear.

mihaibudiu · 2026-09-09T03:36:05Z

+    return planner.getCostFactory().makeCost(cost, 0, 0);
+  }
+
+  @Override public Result implement(EnumerableRelImplementor implementor,


A sequence of comments showing the equivalent generated Java code would make this more readable

Thanks. I've added examples of the generated Java next to the selectors, comparators, and final ieJoin call.

mihaibudiu · 2026-09-09T03:36:56Z

+        left.getCluster().getTypeFactory(), leftType, rightType)
+        && (SqlTypeUtil.isBoolean(leftType)
+            || (SqlTypeUtil.isExactNumeric(leftType)
+                && !SqlTypeName.UNSIGNED_TYPES.contains(typeName))


Is this because Java does not support unsigned types natively?

Thanks! Ordinary unsigned comparisons fail during code generation because overloads such as SqlFunctions.lt(org.joou.UInteger, org.joou.UInteger) are missing. I'd prefer to address this in a separate change.

mihaibudiu · 2026-09-09T03:46:11Z

+        RexProgram.create(ieJoin.getRowType(),
+            rexBuilder.identityProjects(ieJoin.getRowType()), residual,
+            ieJoin.getRowType(), rexBuilder);
+    return EnumerableCalc.create(ieJoin, program);


So the left-over conjunctions are applied in a subsequent filter - and yet, this is still more efficient than the original join?

Yes, this rule adds a Calc above IEJoin to check the remaining conditions. IEJoin finds candidates through sorting and bitmap scans. Whether this is faster depends on how many pairs pass the first two conditions. Thanks!

mihaibudiu · 2026-09-09T03:48:54Z

+      new JavaCollation(SqlCollation.Coercibility.IMPLICIT, Locale.US,
+          Util.getDefaultCharset(), Collator.PRIMARY);
+
+  @Test void ieJoin() {


It's not easy to review these tests.
You can perhaps add a comment to each explaining why you think the output is correct (i.e., has been validated using a different implementation), or why you think the plan has the produced shape.

I've replaced the fixed 600 with a count from a simple Java nested loop over the same data. I've also added comments for the cases.

mihaibudiu · 2026-09-09T03:54:39Z

+ * entries are sorted by each key. A right entry satisfies predicate 1 exactly
+ * when it follows a left entry in {@code firstOrder}, and predicate 2 exactly
+ * when it precedes that left entry in {@code secondOrder}. Sort directions and
+ * tie-breaking depend on the operators; see {@link #entryComparator}.


I see, so you sort twice, once by each compared field?

Yes, that's what I meant: one sort for each comparison. Each has its own sort direction and ordering of equal keys.

mihaibudiu · 2026-09-09T03:55:00Z

+ * entries are sorted by each key. A right entry satisfies predicate 1 exactly
+ * when it follows a left entry in {@code firstOrder}, and predicate 2 exactly
+ * when it precedes that left entry in {@code secondOrder}. Sort directions and
+ * tie-breaking depend on the operators; see {@link #entryComparator}.


what happens if the same compared field is used in both comparisons?

Thanks! Both comparisons can use the same field. The current implementation still sorts twice and doesn't reuse the ordering. I've added tests for this case.

sonarqubecloud · 2026-09-10T15:38:57Z

Quality Gate passed

Issues
19 New issues
0 Accepted issues

Measures
0 Security Hotspots
97.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

zzwqqq · 2026-09-10T15:58:01Z

Why doesn't this work for other types, like DOUBLE?

Thanks! I haven't checked every excluded type yet. For DOUBLE, sorting and SQL comparisons disagree on NaN and signed zero. Unsigned comparisons also fail during code generation. Those are the issues I've confirmed so far.

zzwqqq marked this pull request as draft August 30, 2026 09:30

zzwqqq force-pushed the iejoin branch from 9962f62 to 171d050 Compare September 1, 2026 15:16

zzwqqq marked this pull request as ready for review September 1, 2026 15:18

zzwqqq requested review from mihaibudiu and tjbanghart September 5, 2026 07:39

mihaibudiu reviewed Sep 6, 2026

View reviewed changes

mihaibudiu approved these changes Sep 9, 2026

View reviewed changes

mihaibudiu added the LGTM-will-merge-soon Overall PR looks OK. Only minor things left. label Sep 12, 2026

[CALCITE-7755] Support IEJoin for inequality joins

640e8ba

zzwqqq force-pushed the iejoin branch from 0ba9eea to 640e8ba Compare September 13, 2026 13:14

Uh oh!

Conversation

zzwqqq commented Aug 30, 2026

Jira Link

Changes Proposed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zzwqqq Sep 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mihaibudiu commented Sep 6, 2026

Uh oh!

zzwqqq commented Sep 6, 2026

Uh oh!

mihaibudiu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Sep 10, 2026

Quality Gate passed

Uh oh!

zzwqqq commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zzwqqq Sep 6, 2026 •

edited

Loading