Jackson 3: FAIL_ON_TRAILING_TOKENS default flip not compensated in main-source parsers (JSONataUtils.loadJSONFile + CLI)

[wnm3]

Summary

Jackson 3 flips the FAIL_ON_TRAILING_TOKENS deserialization default from false (Jackson 2) to true. During the migration (#430, PR #431) this leniency was restored on exactly one test mapper (src/test/java/.../expressions/utils/Utils.java:83) but was not restored on the main‑source parse paths, so those became stricter than under Jackson 2. Content after the first parsed value (trailing tokens / trailing typos / concatenated values) that Jackson 2 silently ignored now throws at runtime.

The core public evaluate path is unaffected (it takes a pre‑parsed JsonNode), which is why the test suite stayed green. The exposure is in the file/string loaders below.

Where

Main source (production / public API)

• src/main/java/com/api/jsonata4java/JSONataUtils.java:374 — loadJSONFile uses new ObjectMapper().readValue(br, Object.class). This is a public utility used to load .json files. A file that loaded fine under 2.x (trailing whitespace‑separated token, two concatenated values, etc.) now throws a trailing‑token error. Highest impact.
• src/main/java/com/api/jsonata4java/Tester.java:123 — the mapper was faithfully rebuilt for ESCAPE_NON_ASCII but does not disable FAIL_ON_TRAILING_TOKENS. Clearest evidence the flag flip was known (fixed in test Utils) but applied inconsistently. CLI tool, lower impact.
• src/main/java/com/api/jsonata4java/TesterTimeBox.java:121 — new ObjectMapper() for readTree. CLI tool.
• src/main/java/com/api/jsonata4java/testerui/TesterUI.java and Test.java — same pattern; demo/UI, low impact.

Test source (latent inconsistency)

• src/test/java/.../expressions/utils/Utils.java:76 (getJson) and :247 (toJson) still use bare new ObjectMapper() while the sibling mapper field (line 83) disables FAIL_ON_TRAILING_TOKENS. Same file, divergent parsing. Currently latent (both helpers are unreferenced), but it undermines the "parse identically to Jackson 2" intent stated in the line‑83 comment.

Verified

FAIL_ON_TRAILING_TOKENS defaults to true in jackson‑databind 3.2.0 (bytecode: iconst_1 in the enum init) vs false in Jackson 2. Only Utils.java:83 compensates.

Suggested fix

Introduce a single shared "Jackson‑2‑compatible" mapper factory (disabling FAIL_ON_TRAILING_TOKENS, and centralizing ESCAPE_NON_ASCII etc.) and route all parse sites through it — at minimum JSONataUtils.loadJSONFile, and the Utils.java getJson/toJson helpers, so parsing behavior no longer diverges by call site. This also addresses the duplicated builder configuration noted during review (identical JsonMapper.builder()... blocks now appear in AgnosticTestSuite and Tester).

[wnm3]

Resolved by design: keep the public loadJSONFile (and the other main-source parsers) strict.

Rationale — this project is a functional port of jsonata.js, and the JS reference parses input with JSON.parse, which rejects trailing tokens after the first value. So Jackson 3's stricter FAIL_ON_TRAILING_TOKENS=true default is the functionally-compatible behavior; restoring Jackson 2's leniency on the public API would make the port more permissive than the reference. No compat shim will be added.

The one lenient mapper (src/test/java/.../expressions/utils/Utils.java:83, which disables FAIL_ON_TRAILING_TOKENS) is retained deliberately: it exists only to keep a long-standing test-data typo ("[\"h11\", \"h21\"]]" in BasicExpressionsTests) parsing byte-for-byte, so no test assertion had to change during the migration. That is an intentional test-only exception, not an inconsistency to reconcile.

Closing as won't-fix (working as intended).