Skip to content

Render map wikitext and read GeoJSON pages with the page's parser - #951

Merged
JeroenDeDauw merged 1 commit into
masterfrom
fix/render-on-page-parser
Oct 15, 2026
Merged

JeroenDeDauw merged 1 commit into
masterfrom
fix/render-on-page-parser

Conversation

@JeroenDeDauw

@JeroenDeDauw JeroenDeDauw commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Maps rendered map wikitext, such as marker text and popup templates, and read GeoJSON pages outside the parse of the
page the map is on:

  • Marker text, inline labels and shape text went through Parser::parse() on a copy of the page's parser, or of
    MediaWiki's main parser for the semantic map formats. What that parse recorded, such as templates, links, categories
    and used parser options, went to an output that Maps discarded. parse() also moved the usage tracking of the shared
    parser options onto that output, so options used after a map no longer varied the page's parser cache.
  • The semantic map formats expanded template= popups on a parser of their own, with the requesting user's parser
    options, whose use the page's output did not record. So the popups in a cached render depended on whoever caused
    that render.
  • GeoJSON pages were read from the revision store, not through the parser.

So editing a template or GeoJSON page that a map uses did not refresh the page, links in map wikitext were missing from
the link tables, and readers could get a page with a map rendered for other settings, such as another interface
language. The template and current-revision callbacks of the parser options, which extensions use to decide which pages
a render may include, could not vary the page's cache by what they decided for marker text, applied to template=
popups for the requesting user, and did not apply to GeoJSON pages at all.

Maps now parses each piece of map wikitext with a copy of the page parser's options, on a parser that each map gets from
the parser factory and that starts a fresh parse for each text. The semantic map formats get the page's parser from
Semantic MediaWiki's copyParser(): for #ask and #show, the parser running the query. A parse per text keeps
rendering time linear in the number of markers: parsing each text on the page's own parser makes core render every link
collected so far again, taking 5.8 s for 800 linked markers. A copy of the page's parser would share the state that
extensions keep on it, which starting a parse resets: on MediaWiki 1.43 and 1.44, the page's own TemplateStyles styles
then went missing after a map that used the same styles.

Core's mergeInternalMetaDataFrom() and mergeTrackingMetaDataFrom() then add what the parse recorded to the page's
output: used options, flags, templates, links, categories and extension data sets, such as the pages a callback left
out. The parse's head items, config vars, modules and styles go along through mergeHtmlMetaDataFrom() of an output
that holds only these, so the title, indicators, table of contents and index policy stay the page's. Left out is what
the page decides: page properties and the flags of behavior switches, such as the default sort key and the index policy,
and extension data without a merge strategy, which the merge would replace. That is where extensions such as Semantic
MediaWiki, PageImages, Widgets and Kartographer collect data about the page. A ParserOutput hands over all of its
extension data only in its serialization, so Maps reads the keys from there. MediaWiki 1.46 marks the three merge
methods @internal, as used only by RevisionRenderer, but no public method adds one output to another.

Strip markers in map wikitext belong to the page's parser, so their content goes along to the parser of the text, each
as its own kind, also when nested in another. Tags that parse wikitext, such as galleries, leave link placeholders in
their HTML that only the page's parser can fill in, so the page's parser renders such strip markers with
recursiveTagParseFully() first. When the page's parser only preprocesses, as it does for parser functions under
Parsoid, extension tags in map wikitext are left unprocessed, so Maps puts their wikitext back and parses the map
wikitext in HTML mode; for MediaWiki 1.44 and later, see the last item below. The Widgets extension fills in its widgets
when the parse of the page ends, by which time the markers of a text sit in the map's data, so Maps fills in the widgets
of each text, and those the page expanded in the arguments of the map, before the text goes into the map's data.

GeoJSON pages are read as Scribunto's mw.title:getContent() reads pages, with the page parser's
fetchCurrentRevisionRecordOfTitle(), so the current-revision callback applies to them. Every existing GeoJSON page
Maps reads is recorded as a template of the page, even when a callback refuses it: the link tables come from one render,
but an edit of the GeoJSON page must also reach the renders that showed it.

On Special:Ask and Special:CargoQuery, which nothing caches, Maps uses a parser of its own for the requesting user, as
the semantic map formats did for popups before. Where it gets no parser that is parsing, as when Semantic MediaWiki
hands over one that never parsed or whose parse is over, when an older Cargo hands over a ParserOutput, or when a
caller of MapsFactory::newGeoJsonFetcher() passes none, it parses for an anonymous reader, because it cannot tell
whether the output ends up in a render cached for everyone.

Visible changes:

  • Categories and language links in map wikitext and popup templates now apply to the page, so category queries, such
    as those of Semantic MediaWiki, find it. Maps applied the categories of popup templates before 12.1.0, when it
    expanded them on MediaWiki's main parser.
  • {{PAGENAME}} and relative links in popup templates refer to the page instead of Badtitle/Parser
  • <nowiki>, <pre> and other tags in {{#display_map:}} map wikitext render instead of showing as strip markers,
    except under Parsoid on MediaWiki 1.44 and later, as described below
  • On MediaWiki 1.43 and 1.44, the page's own TemplateStyles styles no longer go missing after a <display_map> whose
    marker text uses the same styles
  • The content of references in <display_map> marker text shows nowhere, as hooks for the end of a page's parse no
    longer run for each text and so the popup gets no reference list. References in popup templates are numbered per
    popup rather than across the map. The page itself numbers and lists those in {{#display_map:}}
  • Widgets render in {{#display_map:}} marker text and in the popups of semantic maps, where they showed as raw
    markers
  • With Semantic MediaWiki 7.0 and older, charts and other query results above a map whose markers have a title or text
    keep their data, scripts and styles, which the parse of map wikitext used to take

Rendering {{#display_map:}} takes as long as on master. Seconds on MediaWiki 1.43:

Markers Each links a page On master No links On master
100 0.05 0.05 0.04 0.04
400 0.18 0.21 0.15 0.17
800 0.38 0.38 0.31 0.31
6400 3.88 3.70 2.66 2.80

1600 markers after 300 Semantic MediaWiki subobjects took 1.11 s, against 1.19 s on master.

Popups of the semantic map formats, which master expanded on one parse state, now get a parse each. Seconds for 500
popups, with LuaSandbox:

Popup text Branch Master
Links a page 0.12 1.11
No links 0.07 0.04
Calls a Lua module 1.41 0.12

On master, the time for linking popups grows with the square of their number: 2000 took 17.7 s. Lua starts anew for
each popup, as it already did for marker text on master, and each popup has the limits of a parse of its own, including
Lua's CPU time limit, so a large map whose popups run heavy Lua can reach the request timeout.

New tests cover each problem above. Compared before and after with a script, the output of 15 representative maps is
otherwise the same: 14 on a page, rendered by the legacy parser and by Parsoid, and one on Special:Ask. The Cargo
format, which uses the parser only to read GeoJSON pages, is unchecked: Cargo was not installed.

CI now also runs on MediaWiki 1.46 and, allowed to fail, on MediaWiki master, which renders with Parsoid by default.
Their composer-generated PHPUnit configuration fails tests without a @covers annotation, which many Maps tests lack,
so these jobs relax that, as core's former tests/phpunit/phpunit.php entry point did.

Public method signatures change: MappingService::newMapDataFromProcessingResult() and newMapDataFromParameters(),
CargoOutputBuilder::buildOutputFromCargoData(), QueryHandler::setTemplate() and the GeoJsonFetcher constructor now
take the parser rendering the page, the WikitextParser constructor also takes a ParserFactory, and MapsFactory
gains getRenderingParser(), newRequestParser() and newWikitextParser(). None of these has a known caller outside
Maps. The one Maps method that Semantic Result Formats calls, MapsFactory::newGeoJsonFetcher(), takes the parser
optionally. Hence the release notes list this under 15.1.0 rather than a new major version.

Considered, omitted:

  • The MediaWiki:Maps page: site configuration, read once per request and documented to take effect at the next parse
  • Icons, image layers and overlays, and KML files: Maps looks up their file URLs, not page content
  • A dependency on GeoJSON pages that do not exist: URL sources would then show up as wanted pages. Creating such a page
    still does not refresh the maps that name it
  • Reading GeoJSON pages through the template callback, as transclusion does: it follows redirects and hands over text
    rather than the revision, whose content model and ID the GeoJSON editor needs
  • Page properties, the display title, indicators and behavior switches in map wikitext: as before, they do not apply to
    the page, apart from the tracking categories some switches add, such as for __NOINDEX__. The page expands the
    arguments of {{#display_map:}} itself, so a {{DEFAULTSORT}} there still applies
  • Semantic MediaWiki annotations in map wikitext: as before, they do not annotate the page
  • The <!--MW-PAGEIMAGES-CANDIDATE-N--> comments that PageImages adds to images stay in popup HTML, where nothing shows
    them
  • Extra parser cache keys that extensions add with ParserOptions::addExtraKey() while parsing a text: the parse uses a
    copy of the page's options, so the keys do not vary the page's cache
  • Moving the hand-over of strip markers to a pull request of its own: parsing a text with recursiveTagParseFully()
    rather than parse() stops defusing the page's strip markers, so this pull request has to hand them over, and the
    Parsoid sanitization lives in the same step
  • Semantic Compound Queries and other callers that get whichever parser Semantic MediaWiki last held: while that parser
    is parsing another page, what map wikitext records lands in that page's output, where before it went to a discarded
    copy
  • Semantic Result Formats' filtered format, which calls newGeoJsonFetcher() without a parser and so reads GeoJSON
    pages as an anonymous reader: passing its parser is a change in that extension
  • Extension tags in {{#display_map:}} map wikitext under Parsoid on MediaWiki 1.44 and later: core leaves them for
    Parsoid to run, so <nowiki> content is parsed as wikitext, where before it showed as a strip marker, and other tags
    still show as strip markers, as before. Telling these strip markers apart takes StripState methods that MediaWiki
    1.43 lacks

AI-authored — Claude Code, Opus 5.5 (max); detailed spec from @JeroenDeDauw, rendering reworked once on an orchestrator decision and revised after AI code, test and prose reviews; diff not yet human-reviewed; regression tests failed before each fix, PHPUnit with SMW 7, phpcs and PHPStan pass locally on MediaWiki 1.43, map output and timings compared with master on the legacy parser and Parsoid, CI green with SMW 6 and 7.

@JeroenDeDauw
JeroenDeDauw marked this pull request as ready for review October 15, 2026 16:53
Maps rendered map wikitext, such as marker text and popup templates, and read GeoJSON pages outside the parse of the
page the map is on:

* Marker text, inline labels and shape text went through `Parser::parse()` on a copy of the page's parser, or of
  MediaWiki's main parser for the semantic map formats. What that parse recorded, such as templates, links, categories
  and used parser options, went to an output that Maps discarded. `parse()` also moved the usage tracking of the shared
  parser options onto that output, so options used after a map no longer varied the page's parser cache.
* The semantic map formats expanded `template=` popups on a parser of their own, with the requesting user's parser
  options, whose use the page's output did not record. So the popups in a cached render depended on whoever caused
  that render.
* GeoJSON pages were read from the revision store, not through the parser.

So editing a template or GeoJSON page that a map uses did not refresh the page, links in map wikitext were missing from
the link tables, and readers could get a page with a map rendered for other settings, such as another interface
language. The template and current-revision callbacks of the parser options, which extensions use to decide which pages
a render may include, could not vary the page's cache by what they decided for marker text, applied to `template=`
popups for the requesting user, and did not apply to GeoJSON pages at all.

Maps now parses each piece of map wikitext with a copy of the page parser's options, on a parser that each map gets from
the parser factory and that starts a fresh parse for each text. The semantic map formats get the page's parser from
Semantic MediaWiki's `copyParser()`: for `#ask` and `#show`, the parser running the query. A parse per text keeps
rendering time linear in the number of markers: parsing each text on the page's own parser makes core render every link
collected so far again, taking 5.8 s for 800 linked markers. A copy of the page's parser would share the state that
extensions keep on it, which starting a parse resets: on MediaWiki 1.43 and 1.44, the page's own TemplateStyles styles
then went missing after a map that used the same styles.

Core's `mergeInternalMetaDataFrom()` and `mergeTrackingMetaDataFrom()` then add what the parse recorded to the page's
output: used options, flags, templates, links, categories and extension data sets, such as the pages a callback left
out. The parse's head items, config vars, modules and styles go along through `mergeHtmlMetaDataFrom()` of an output
that holds only these, so the title, indicators, table of contents and index policy stay the page's. Left out is what
the page decides: page properties and the flags of behavior switches, such as the default sort key and the index policy,
and extension data without a merge strategy, which the merge would replace. That is where extensions such as Semantic
MediaWiki, PageImages, Widgets and Kartographer collect data about the page. A `ParserOutput` hands over all of its
extension data only in its serialization, so Maps reads the keys from there. MediaWiki 1.46 marks the three merge
methods `@internal`, as used only by `RevisionRenderer`, but no public method adds one output to another.

Strip markers in map wikitext belong to the page's parser, so their content goes along to the parser of the text, each
as its own kind, also when nested in another. Tags that parse wikitext, such as galleries, leave link placeholders in
their HTML that only the page's parser can fill in, so the page's parser renders such strip markers with
`recursiveTagParseFully()` first. When the page's parser only preprocesses, as it does for parser functions under
Parsoid, extension tags in map wikitext are left unprocessed, so Maps puts their wikitext back and parses the map
wikitext in HTML mode; for MediaWiki 1.44 and later, see the last item below. The Widgets extension fills in its widgets
when the parse of the page ends, by which time the markers of a text sit in the map's data, so Maps fills in the widgets
of each text, and those the page expanded in the arguments of the map, before the text goes into the map's data.

GeoJSON pages are read as Scribunto's `mw.title:getContent()` reads pages, with the page parser's
`fetchCurrentRevisionRecordOfTitle()`, so the current-revision callback applies to them. Every existing GeoJSON page
Maps reads is recorded as a template of the page, even when a callback refuses it: the link tables come from one render,
but an edit of the GeoJSON page must also reach the renders that showed it.

On Special:Ask and Special:CargoQuery, which nothing caches, Maps uses a parser of its own for the requesting user, as
the semantic map formats did for popups before. Where it gets no parser that is parsing, as when Semantic MediaWiki
hands over one that never parsed or whose parse is over, when an older Cargo hands over a `ParserOutput`, or when a
caller of `MapsFactory::newGeoJsonFetcher()` passes none, it parses for an anonymous reader, because it cannot tell
whether the output ends up in a render cached for everyone.

Visible changes:
* Categories and language links in map wikitext and popup templates now apply to the page, so category queries, such
  as those of Semantic MediaWiki, find it. Maps applied the categories of popup templates before 12.1.0, when it
  expanded them on MediaWiki's main parser.
* `{{PAGENAME}}` and relative links in popup templates refer to the page instead of `Badtitle/Parser`
* `<nowiki>`, `<pre>` and other tags in `{{#display_map:}}` map wikitext render instead of showing as strip markers,
  except under Parsoid on MediaWiki 1.44 and later, as described below
* On MediaWiki 1.43 and 1.44, the page's own TemplateStyles styles no longer go missing after a `<display_map>` whose
  marker text uses the same styles
* The content of references in `<display_map>` marker text shows nowhere, as hooks for the end of a page's parse no
  longer run for each text and so the popup gets no reference list. References in popup templates are numbered per
  popup rather than across the map. The page itself numbers and lists those in `{{#display_map:}}`
* Widgets render in `{{#display_map:}}` marker text and in the popups of semantic maps, where they showed as raw
  markers
* With Semantic MediaWiki 7.0 and older, charts and other query results above a map whose markers have a title or text
  keep their data, scripts and styles, which the parse of map wikitext used to take

Rendering `{{#display_map:}}` takes as long as on master. Seconds on MediaWiki 1.43:

| Markers | Each links a page | On master | No links | On master |
|--------:|------------------:|----------:|---------:|----------:|
|     100 |              0.05 |      0.05 |     0.04 |      0.04 |
|     400 |              0.18 |      0.21 |     0.15 |      0.17 |
|     800 |              0.38 |      0.38 |     0.31 |      0.31 |
|    6400 |              3.88 |      3.70 |     2.66 |      2.80 |

1600 markers after 300 Semantic MediaWiki subobjects took 1.11 s, against 1.19 s on master.

Popups of the semantic map formats, which master expanded on one parse state, now get a parse each. Seconds for 500
popups, with LuaSandbox:

| Popup text                    | Branch | Master |
|-------------------------------|-------:|-------:|
| Links a page                  |   0.12 |   1.11 |
| No links                      |   0.07 |   0.04 |
| Calls a Lua module            |   1.41 |   0.12 |

On master, the time for linking popups grows with the square of their number: 2000 took 17.7 s. Lua starts anew for
each popup, as it already did for marker text on master, and each popup has the limits of a parse of its own, including
Lua's CPU time limit, so a large map whose popups run heavy Lua can reach the request timeout.

New tests cover each problem above. Compared before and after with a script, the output of 15 representative maps is
otherwise the same: 14 on a page, rendered by the legacy parser and by Parsoid, and one on Special:Ask. The Cargo
format, which uses the parser only to read GeoJSON pages, is unchecked: Cargo was not installed.

CI now also runs on MediaWiki 1.46 and, allowed to fail, on MediaWiki master, which renders with Parsoid by default.
Their composer-generated PHPUnit configuration fails tests without a `@covers` annotation, which many Maps tests lack,
so these jobs relax that, as core's former `tests/phpunit/phpunit.php` entry point did.

Public method signatures change: `MappingService::newMapDataFromProcessingResult()` and `newMapDataFromParameters()`,
`CargoOutputBuilder::buildOutputFromCargoData()`, `QueryHandler::setTemplate()` and the `GeoJsonFetcher` constructor now
take the parser rendering the page, the `WikitextParser` constructor also takes a `ParserFactory`, and `MapsFactory`
gains `getRenderingParser()`, `newRequestParser()` and `newWikitextParser()`. None of these has a known caller outside
Maps. The one Maps method that Semantic Result Formats calls, `MapsFactory::newGeoJsonFetcher()`, takes the parser
optionally. Hence the release notes list this under 15.1.0 rather than a new major version.

Considered, omitted:
* The `MediaWiki:Maps` page: site configuration, read once per request and documented to take effect at the next parse
* Icons, image layers and overlays, and KML files: Maps looks up their file URLs, not page content
* A dependency on GeoJSON pages that do not exist: URL sources would then show up as wanted pages. Creating such a page
  still does not refresh the maps that name it
* Reading GeoJSON pages through the template callback, as transclusion does: it follows redirects and hands over text
  rather than the revision, whose content model and ID the GeoJSON editor needs
* Page properties, the display title, indicators and behavior switches in map wikitext: as before, they do not apply to
  the page, apart from the tracking categories some switches add, such as for `__NOINDEX__`. The page expands the
  arguments of `{{#display_map:}}` itself, so a `{{DEFAULTSORT}}` there still applies
* Semantic MediaWiki annotations in map wikitext: as before, they do not annotate the page
* The `<!--MW-PAGEIMAGES-CANDIDATE-N-->` comments that PageImages adds to images stay in popup HTML, where nothing shows
  them
* Extra parser cache keys that extensions add with `ParserOptions::addExtraKey()` while parsing a text: the parse uses a
  copy of the page's options, so the keys do not vary the page's cache
* Moving the hand-over of strip markers to a pull request of its own: parsing a text with `recursiveTagParseFully()`
  rather than `parse()` stops defusing the page's strip markers, so this pull request has to hand them over, and the
  Parsoid sanitization lives in the same step
* Semantic Compound Queries and other callers that get whichever parser Semantic MediaWiki last held: while that parser
  is parsing another page, what map wikitext records lands in that page's output, where before it went to a discarded
  copy
* Semantic Result Formats' filtered format, which calls `newGeoJsonFetcher()` without a parser and so reads GeoJSON
  pages as an anonymous reader: passing its parser is a change in that extension
* Extension tags in `{{#display_map:}}` map wikitext under Parsoid on MediaWiki 1.44 and later: core leaves them for
  Parsoid to run, so `<nowiki>` content is parsed as wikitext, where before it showed as a strip marker, and other tags
  still show as strip markers, as before. Telling these strip markers apart takes `StripState` methods that MediaWiki
  1.43 lacks

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@JeroenDeDauw
JeroenDeDauw force-pushed the fix/render-on-page-parser branch from ff516e5 to 0cddc27 Compare October 15, 2026 16:54
@JeroenDeDauw
JeroenDeDauw merged commit 88eb25e into master Oct 15, 2026
10 checks passed
@JeroenDeDauw
JeroenDeDauw deleted the fix/render-on-page-parser branch October 15, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant