[Bug] [Bug] first_value/nth_value return corrupted values and can crash the BE once a window partition spans more than 257 blocks

[hanke580]

Search before asking

• I had searched in the issues and found no similar issues.

Version

Affected: Apache Doris 4.1.4 — BE reports doris-4.1.4-rc04-ad35a140c7f (the newest
released build; images apache/doris:fe-4.1.4 + be-4.1.4, pushed 2026-09-11). Single FE +
single BE, Nereids planner, otherwise default configuration, Linux x86_64.

Also affected: current master. There is no nightly/master Doris image, so I built
apache/doris master commit c4dee4bd5e82c6f0f380e0243cbaf265999e6a6a from source with
apache/doris:build-env-ldb-toolchain-latest and ran the identical battery against a
single-FE/single-BE cluster from that build. The clone's HEAD was verified equal to GitHub's
master HEAD (2026-09-17T03:34:48Z). BE reports doris-0.0.0-Unknown because a local build
carries no version stamp; the shipped default batch_size is 8160 there too.

master reproduces every measurement identically to 4.1.4 — not approximately, the same
counts row for row:

measurement	4.1.4	master c4dee4bd
first_value(varchar) wrong @ batch_size 4 / 2 / 1	4 / 10 / 9	4 / 10 / 9
longest bogus string @ 4 / 2 / 1	229 / 525 / 772	229 / 525 / 772
first_value(int) wrong @ batch_size 38 / 37 / 4 / 1	392 / 908 / 15 888 / 17 180	392 / 908 / 15 888 / 17 180
onset rule at partition size 10 000 / 5 000 / 2 000	bs 38 / 19 / 7	38 / 19 / 7
2 560 000-row partition @ default batch_size 8160	454 720 wrong	454 720 wrong
1 280 000-row partition @ default batch_size 8160	0	0
nth_value string length	total_length=4294967303	total_length=4294967296 (2³², message now also prints limit=4294967295)

and master crashed the BE during the run. Because it is a locally built binary the trace
carries full source lines, which is the clearest statement of the bug available:

*** SIGSEGV invalid permissions for mapped object (@0x7f15c87fd000) received by PID 2333 ***
 2# memcpy at ./be/build_Release/../src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
 3# doris::ColumnStr<unsigned int>::insert_from(doris::IColumn const&, unsigned long)
      at ../src/core/column/column_string.h:188
 4# doris::IAggregateFunction::insert_result_into_range(char const*, doris::IColumn&, unsigned long, unsigned long) const
      at ../src/exprs/aggregate/aggregate_function.h
 5# doris::AnalyticSinkLocalState::_insert_result_info(long, long)
      at ./be/src/exec/operator/analytic_sink_operator.cpp:404
 6# doris::AnalyticSinkLocalState::_get_next_for_unbounded_rows(long, long)
      at ./be/src/exec/operator/analytic_sink_operator.cpp:248
 8# doris::AnalyticSinkOperatorX::sink_impl(doris::RuntimeState*, doris::Block*, bool)
      at ./be/src/exec/operator/analytic_sink_operator.cpp:759

A memcpy inside ColumnStr::insert_from, reached from the analytic sink writing out an
aggregate's saved value — i.e. copying a string through an offset that no longer describes the
row it was saved for.

No fix is in flight: the most recent commit touching analytic_sink_operator.cpp is #67274
(2026-08-31) and the most recent touching aggregate_function_window.h is #64864 (2026-06-26);
neither addresses the saved-pointer invalidation.

What's Wrong?

first_value() over a window silently returns values that are not in the window — and on a
VARCHAR column, values that are not in the table at all. On the same data the BE can also
abort the query with a bogus 4 GB string length, or crash outright.

The trigger is that the analytic operator evicts input rows it believes are no longer needed,
while first_value is still holding a raw pointer plus an absolute row offset into exactly
the columns being evicted.

1. first_value(varchar) returns a string that is nowhere in the table

With PARTITION BY c, first_value(c) must equal the row's own c — every partition contains
one distinct value, so this is an exact oracle needing no reference engine. 20 000 rows, every
value at most 8 characters:

batch_size	rows wrong	longest bogus value returned
65535	0	—
8160 (shipped default)	0	—
8	0	—
4	4	229 chars
2	10	525 chars
1	9	772 chars

A sample row: c = 'xyz', so the correct answer is 'xyz', and Doris returns

azczazaczcbczcbczcbczyaxabxxaaxbbxabaxabcayabxacazayxaxcxyzxaxyxxxx…   (229 characters)

That is a run of adjacent values read out of the string column through a wrong offset/length
pair — the correct 3-character value is visible inside it.

2. first_value(int) is wrong for most of the table

Same shape on an INT column, ground truth computed in Python (PARTITION BY e with e boolean,
so two partitions of ~10 000 rows):

batch_size	rows wrong (of 20 000)
65535 / 8160 / 64 / 40	0
39	0
38	392
37	908
4	15 888 (79%)
1	17 180 (86%)

No warning, no error — the query simply returns wrong numbers for most rows.

3. It reproduces at the shipped batch_size on an ordinary table

batch_size only makes it cheap to demonstrate. What actually matters is the partition size in
blocks, so the same defect is reached with batch_size left completely alone (shipped default
8160) by making one partition large — the threshold being 257 * 8160 = 2 097 120 rows.
Single partition, nothing configured:

rows in the partition	wrong at default batch_size (8160)	wrong at batch_size = 65535
160 000 – 1 280 000	0	0
2 560 000	454 720 (18%)	0

The boundary lands exactly where the 257 * batch_size rule predicts (1 280 000 < 2 097 120 <
2 560 000), and raising batch_size to 65535 moves the threshold out of reach again, which
isolates the block count as the cause.

One caveat worth knowing when reproducing: the block count is produced by the scan, so it
depends on how the table is physically laid out, not only on batch_size. On a table I had
built by repeated INSERT INTO … SELECT doubling (many small rowsets) the same 2 560 000-row
partition was wrong at batch_size = 65535 as well, where a table of the same size written by a
single INSERT … SELECT was clean at 65535. So a fragmented table reaches 257 blocks with fewer
rows, and raising batch_size does not reliably protect it.

4. The exact trigger

The onset is not fuzzy. Corruption begins exactly when

partition_size  >  257 * batch_size

which is block_num = 256 in _remove_unused_rows(), plus the + 1 in its guard. Verified by
fixing the partition size and walking batch_size across the predicted boundary:

partition size	predicted onset	bs = pred+1	bs = pred	bs = pred−1
10 000	bs ≤ 38	0 wrong	392 wrong	908 wrong
5 000	bs ≤ 19	0 wrong	392 wrong	1 424 wrong
2 000	bs ≤ 7	0 wrong	1 940 wrong	4 520 wrong

Three for three, on the row exactly.

5. The same corruption as an error, and as a crash

nth_value(varchar, 3) over ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING, 20 000 rows,
batch_size = 4:

errCode = 2, detailMessage = [E-3113] string column length is too large:
total_length=4294967303, element_number=4, rows=0

4294967303 is 2³² + 7 — the string length arithmetic wrapped a 32-bit counter, which is the
same wrong offset/length pair that produces the 229-character first_value answers.

The same shape took the BE down on both builds — on 4.1.4 at batch_size = 1, and on the
master build during the battery. Stack from the 4.1.4 BE log (the master one, with full source
lines, is in the Version section):

2#  doris::ColumnStr<unsigned int>::insert_from(doris::IColumn const&, unsigned long)
       at be/src/core/column/column_string.h:71
3#  doris::IAggregateFunction::insert_result_into_range(char const*, doris::IColumn&, unsigned long, unsigned long) const
       at be/src/exprs/aggregate/aggregate_function.h:252
4#  doris::AnalyticSinkLocalState::_insert_result_info(long, long)
       at be/src/exec/operator/analytic_sink_operator.cpp:406
5#  doris::AnalyticSinkLocalState::_get_next_for_unbounded_rows(long, long)
       at be/src/exec/operator/analytic_sink_operator.cpp:250
6#  doris::AnalyticSinkLocalState::_execute_impl(doris::RuntimeState*)
7#  doris::AnalyticSinkOperatorX::sink_impl(doris::RuntimeState*, doris::Block*, bool)

Frame 2 is the aggregate's saved value being copied out: to.insert_from(*_ptr, _offset). It is
reading through a pointer/offset that no longer describes the row it was saved for.

What You Expected?

first_value(c) OVER (PARTITION BY c …) must return the row's own c for every row, and
first_value(a) must return the first a of the partition in window order — regardless of
batch_size, partition size, or whether the operator decided to release memory mid-partition.
The answer must not depend on how the input was split into vectorised blocks.

At minimum, a query must not silently return values that do not exist in the input, and must not
crash the backend.

How to Reproduce?

Single FE + single BE, default configuration:

docker network create --subnet=172.30.0.0/24 doris-net
docker run -d --name fe --network doris-net --ip 172.30.0.2 \
  -e FE_SERVERS="fe1:172.30.0.2:9010" -e FE_ID=1 -p 9030:9030 apache/doris:fe-4.1.4
docker run -d --name be --network doris-net --ip 172.30.0.3 \
  -e FE_SERVERS="fe1:172.30.0.2:9010" -e BE_ADDR="172.30.0.3:9050" apache/doris:be-4.1.4

A. Smallest form — exact oracle, no reference engine needed.
Load 20 000 rows of (id INT, c VARCHAR(255)) where c is a random string of at most 8
characters, then:

SET batch_size = 4;                     -- 20 000 rows / 4 = 5 000 blocks per scan
SELECT id, c,
       first_value(c) OVER (PARTITION BY c ORDER BY c, id
                            ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS w
FROM t
HAVING w <> c;                          -- must be empty; returns rows whose w is 229 chars long

Because the window partitions by c, w can only be c. Any returned row is a wrong answer.
SET batch_size = 65535 on the same table returns nothing, confirming the data is fine.

B. At the shipped batch_size, no session variable touched.
Build (id INT, a INT, g BIGINT, e INT) with e = 0 on every row, so the whole table is one
partition, and grow it past 257 * 8160 = 2 097 120 rows:

-- batch_size left at its shipped default of 8160
SELECT count(*) FROM (
  SELECT id, first_value(a) OVER (PARTITION BY e ORDER BY g, id
                                  ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS w
  FROM big
) s
WHERE w <> (SELECT a FROM big ORDER BY g, id LIMIT 1);
-- 1 280 000 rows -> 0        (below 257*8160)
-- 2 560 000 rows -> 454 720  (must be 0)

SET batch_size = 65535 returns 0 for both, which isolates the block count as the cause.

C. The crash / 4 GB length.
On the 20 000-row varchar table:

SET batch_size = 4;
SELECT id, nth_value(c, 3) OVER (PARTITION BY b % 13 ORDER BY c, id
                                 ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS w
FROM t;
-- [E-3113] string column length is too large: total_length=4294967303

SET batch_size = 1;                      -- took the BE down in my run

A reproduction script with the ground-truth oracles for all of the above is attached to this
report.

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Initial triage: confirmed by code inspection; high-severity correctness and availability issue. The report contains strong deterministic evidence for silent wrong results and a BE crash. I did not independently run the Docker reproduction, but the relevant code at both ad35a140c7f (4.1.4) and c4dee4bd5e82 (reported master) has the ownership violation needed to produce the reported symptoms.

Code-backed mechanism

• Value::set_value() in be/src/exprs/aggregate/aggregate_function_reader_first_last.h saves only an IColumn* plus a row offset. FirstLastData::insert_result_into() later reads that pair with insert_from(*_ptr, _offset). FirstLastData is used by first_value, and NthValueData derives from it.
• The analytic sink appends arguments into the long-lived _agg_input_columns. Before accepting another block, _remove_unused_rows() chooses a 256-block prefix and calls erase(0, remove_rows) on those same columns. It rebases _current_row_position, partition/order boundaries, and queued ends, but it neither copies retained aggregate values nor rebases/invalidates offsets stored inside aggregate state.
• The column object itself remains alive; the defect is more specifically a stale or out-of-range saved row offset after the column contents are compacted. For fixed-width columns, erase() memmoves later values down, explaining deterministic but incorrect integers. For ColumnStr, erase() compacts both chars and offsets; a later out-of-range insert_from() computes a bogus length/offset pair, which is consistent with the 4 GiB length error and the memcpy crash.
• first_value can keep that reference because _has_value makes subsequent calls return without replacing it. The eviction guard added by [fix](exec) Retain sliding window rows during eviction #67274 protects rows required by the next ROWS-frame calculation, but it does not account for references already retained inside an aggregate state.
• The production block_num = 256 and the block_num + 1 guard in _remove_unused_rows() directly explain why the transition is tied to the number of physical input blocks. Increasing batch_size only moves the boundary and is not a correctness guarantee, especially for fragmented input.

Missing/useful follow-up information

The body says a complete reproduction script is attached, but there is currently no attachment or comment. Please upload the deterministic script, including the full DDL (the nth_value example uses column b, which is absent from the abbreviated two-column DDL), data generator/seed, and load procedure. A complete BE fatal-log excerpt with query ID would also be useful for validating the eventual fix, but this is not a blocker for accepting the bug: the supplied versions, measurements, error, and stack are already sufficient for triage.

Recommended next steps

1. Add a focused BE regression in be/test/exec/operator/analytic_sink_operator_test.cpp. Under BE_TEST, the eviction threshold is already one block, so the test can exercise first_value/nth_value across eviction with only a few blocks. Cover at least fixed-width and string inputs, nullable/ignore-null variants where applicable, and assert that eviction actually occurred.
2. Make the retained window result own/copy its value, or introduce an explicit aggregate-state/eviction contract that preserves and rebases every referenced row. Merely changing the 256-block threshold is not a fix. Copying the selected value is the safer direction for first_value/nth_value; retaining the entire referenced prefix could defeat the memory-release purpose for unbounded frames.
3. Audit last_value, lead, and lag as well: their window state uses the same Value/BaseValue pointer-plus-offset representation, although this report does not establish that every one of those functions is reachable with an unsafe lifetime.
4. Validate the fix with the reporter's three oracles, an ASAN build for the string case, default batch_size with a partition exceeding the physical-block threshold, and both master and branch-4.1. A 4.1 backport is warranted because 4.1.4 is affected.
5. Until a fix is available, avoid these window shapes on partitions that may span many physical scan blocks. Raising batch_size may reduce exposure but should not be documented as a safe workaround.

No labels are currently attached. This should be treated as a bug with high correctness/reliability severity due to silent result corruption plus process crash.

Breakwater-GitHub-Analysis-Slot: slot_a09751a013c3