Projection in drop mode computes the dropped attributes with a different case sensitivity at compile time and at runtime

[eugenegujing]

What happened?

The Projection operator's "Drop Option" (isDrop = true) computes "which attributes to remove" in two places that disagree on whether attribute names are case-sensitive.

The descriptor derives the output schema through Schema.remove, which lowercases both sides before comparing:

// common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/projection/ProjectionOpDesc.scala:67-71
        } else {
          attributes.foldLeft(inputSchema) { (schema, attribute) =>
            schema.remove(attribute.getOriginalAttribute)
          }
        }

// common/workflow-core/src/main/scala/org/apache/texera/amber/core/tuple/Schema.scala:170-174
  def remove(attributeNames: Iterable[String]): Schema = {
    val attributesToRemove = attributeNames.map(_.toLowerCase).toSet

The executor computes the kept set with List.diff, which compares names exactly:

// common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/projection/ProjectionOpExec.scala:40-43
    if (desc.isDrop) {
      val allAttribute = tuple.schema.getAttributeNames
      val selectedAttributes = desc.attributes.map(_.getOriginalAttribute)
      val keepAttributes = allAttribute.diff(selectedAttributes)

So for an input schema containing field1 and a drop entry spelled Field1: the descriptor removes field1 from the declared output schema, while the executor's diff does not match it and project() keeps it. A genuinely misspelled attribute fails at plan time (Schema.remove throws on non-existent names), but a merely differently-cased one passes that check and diverges silently at runtime.

The divergence does not reach downstream operators in the current engine: DataProcessor re-projects the executor's MapTupleLike onto the declared port schema by attribute name, which discards the extra field, so the end-to-end output happens to match the declared schema. The inconsistency is internal to the operator, and end-to-end correctness currently depends on that name-based re-projection rather than on the two halves agreeing. The Schema class is otherwise uniformly case-insensitive (getIndex, containsAttribute, add, remove all lowercase before comparing); the executor's raw diff is the outlier.

The executor-side behavior is pinned by the drop-mode spec "match drop names case-sensitively" in ProjectionOpExecSpec, whose comment notes that it diverges from the descriptor.

How to reproduce?

Reproducible at the unit level, against a schema with attributes field1, field2, field3:

1. Configure a ProjectionOpDesc with isDrop = true and a single drop entry spelled FIELD2.
2. Derive the output schema through the descriptor: Schema.remove matches case-insensitively, so the declared output schema is field1, field3.
3. Feed one tuple through ProjectionOpExec.project with the same configuration: the executor's diff matches exactly, finds no attribute named FIELD2, and the returned tuple still contains all three fields — including the field2 the declared schema removed.

The existing spec "match drop names case-sensitively" in ProjectionOpExecSpec runs exactly step 3 and pins its result, so running that spec is the fastest way to see the divergence.

There are no end-to-end steps because the divergence is not observable from a running workflow: creating the case-mismatched configuration requires bypassing the UI in the first place (the attribute picker autofills exact names, so it takes a hand-edited or imported workflow JSON, or an upstream schema whose casing changed after the operator was configured), and even then the engine re-projects the executor's output onto the declared schema before it reaches downstream operators or storage, so the final result looks correct regardless.

Version/Branch

1.3.0-incubating-SNAPSHOT (main)

Commit Hash (Optional)

No response

What browsers are you seeing the problem on?

No response

Relevant log output