✨ Render marked-RST blocks in src-trace directive (#43)

[cpolzer]

Summary

Fixes the gap discussed in #43: the src-trace directive extracted marked-RST blocks (@rst ... @endrst) when get_rst = true was configured, but then silently discarded them. Only one-line needs made it into the doctree.

This PR wires marked-RST rendering into the directive. Support remains opt-in via the existing get_rst toggle — behaviour for projects that do not enable it is unchanged.

Behaviour

When get_rst = true is set for a project, each extracted RST block is parsed inline into the current document via nested_parse_with_titles. Authors keep full control over what the block emits — sphinx-needs directives (.. impl::, .. spec::, ...), cross-references, admonitions, plain paragraphs, etc.

Example marker in C++:

/*
@rst
.. impl:: implement dummy function 1
   :id: IMPL_71
@endrst
*/
void dummy_func1() {}

renders the impl need into the document that hosts the src-trace directive.

Source-page anchor mappings (file_lineno_href) are updated for marked-RST blocks too, so the generated highlighted source page links the marker line back to the hosting document when set_local_url = true.

Changes

• src/sphinx_codelinks/sphinx_extension/directives/src_trace.py — extend render_needs() with a _render_marked_rst() helper.
• docs/source/components/directive.rst — drop the "only supports one-line needs" attention banner; document the opt-in and add a C++ example.
• tests/doc_test/marked_rst_basic/ — new Sphinx fixture with get_rst = true and a C++ source containing a marked-RST impl.
• tests/test_src_trace.py — parametrize test_build_html with the new fixture.
• tests/__snapshots__/test_src_trace/…sphinx_project6…doctree.xml — asserts the Need node is rendered with the correct id and body.

Verification

• pytest: 347 passed (59 snapshots).
• pre-commit run --all-files: green.
• mypy on the touched file: clean.

Open questions for reviewers

• Default toggle. get_rst stays opt-in and remains False by default (also on the Sphinx side). Happy to auto-enable it under the extension if you prefer.
• Anchor target for source pages. Marked-RST blocks may not carry a need id, so the source-page anchor links to docname only (not docname#need-id). Open to a different convention.
• Java / other block-comment languages. Confirmed the marker syntax is language-agnostic once the language is a supported comment_type; adding Java is out of scope for this PR.

Closes #43.

[cpolzer]

@ubmarco @juiwenchen not sure if this helps for the source_to_rst topic?

[PhilipPartsch]

Thanks @cpolzer. Wonderful work

[ubmarco]

Thanks for this — the approach is clean, the per-line StringList.items source mapping is worth a close look, and keeping it opt-in via get_rst is the right call. One real gap needs a decision before merging.

The parser is the host's, not RST's. nested_parse_with_titles(self.state, …) is a thin wrapper over state.nested_parse(..., match_titles=True) (Sphinx source) that parses the StringList with the current document's state. In an .rst file that's docutils RST; in a MyST/.md file it's MyST's state, so the @rst … @endrst block is parsed as Markdown — the same concern Chris raised on #66.

Difference vs. plain nested_parse in Chris's PR: both go through the same host state, but differ in what the content is allowed to be:

• nested_parse_with_titles (✨ Render marked-RST blocks in src-trace directive (#43) #99, whole block) = match_titles=True inside a fresh title-style context. It treats the block as an independent mini-document: section titles are allowed, and title decorations don't need to match the surrounding doc. Its docstring says it's for content coming from a completely different context, such as docstrings — which is exactly this "RST text lifted out of a code comment" case.
• plain nested_parse (Chris's PR, the need body) = match_titles=False. Content is a directive body: no section titles allowed.

Crucially, neither is parser-agnostic — both call state.nested_parse(...) with the host state. In Chris's PR only the directive header is host-agnostic (parse_single_directive() regex in analyse/utils.py); the body still goes through state.nested_parse (sphinx-needs), so in a .md host the body is Markdown too. So #66 is only RST-correct for the header, #99 only in .rst hosts.

Recommendation on the parse call. I'd avoid nested_parse_with_titles here: (a) it's deprecated in Sphinx 8 and the range goes up to Sphinx 9, where it warns; and (b) match_titles=True allows section headings into a container that is a child of the src-trace directive — structurally invalid, since docutils only permits sections under document/section/sidebar, and the documented use case (needs, admonitions, paragraphs) never produces headings anyway. Prefer nested_parse_to_nodes(state, text, source=…, offset=…, allow_section_headings=False) (Sphinx docs). Version check: nested_parse_to_nodes was added in Sphinx 7.4, and sphinx-codelinks already requires sphinx>=7.4,<10, so it's available across the entire supported range — no version gating needed.

And the CLI can't decide either. codelinks has a CLI (codelinks analyse → JSON) that reads markers with no Sphinx build present — we don't interpret the RST at that stage at all, and don't know whether it will ever render in a Sphinx or ubCode scope. So "is this RST" can't be decided at extraction.

My position — this makes it work cleanly: on the Sphinx level, reject marked-RST content when src-trace is not in an RST document: raise a warning and skip extraction. Then "RST-ness" is guaranteed by construction — in an RST host it parses as RST, and we never silently reinterpret it as Markdown. The CLI stays format-agnostic (raw text out); the Sphinx layer enforces the contract.

[cpolzer]

Thanks for this — the approach is clean, the per-line StringList.items source mapping is worth a close look, and keeping it opt-in via get_rst is the right call. One real gap needs a decision before merging.

The parser is the host's, not RST's. nested_parse_with_titles(self.state, …) is a thin wrapper over state.nested_parse(..., match_titles=True) (Sphinx source) that parses the StringList with the current document's state. In an .rst file that's docutils RST; in a MyST/.md file it's MyST's state, so the @rst … @endrst block is parsed as Markdown — the same concern Chris raised on #66.

Difference vs. plain nested_parse in Chris's PR: both go through the same host state, but differ in what the content is allowed to be:

• nested_parse_with_titles (✨ Render marked-RST blocks in src-trace directive (#43) #99, whole block) = match_titles=True inside a fresh title-style context. It treats the block as an independent mini-document: section titles are allowed, and title decorations don't need to match the surrounding doc. Its docstring says it's for content coming from a completely different context, such as docstrings — which is exactly this "RST text lifted out of a code comment" case.
• plain nested_parse (Chris's PR, the need body) = match_titles=False. Content is a directive body: no section titles allowed.

Crucially, neither is parser-agnostic — both call state.nested_parse(...) with the host state. In Chris's PR only the directive header is host-agnostic (parse_single_directive() regex in analyse/utils.py); the body still goes through state.nested_parse (sphinx-needs), so in a .md host the body is Markdown too. So #66 is only RST-correct for the header, #99 only in .rst hosts.

Recommendation on the parse call. I'd avoid nested_parse_with_titles here: (a) it's deprecated in Sphinx 8 and the range goes up to Sphinx 9, where it warns; and (b) match_titles=True allows section headings into a container that is a child of the src-trace directive — structurally invalid, since docutils only permits sections under document/section/sidebar, and the documented use case (needs, admonitions, paragraphs) never produces headings anyway. Prefer nested_parse_to_nodes(state, text, source=…, offset=…, allow_section_headings=False) (Sphinx docs). Version check: nested_parse_to_nodes was added in Sphinx 7.4, and sphinx-codelinks already requires sphinx>=7.4,<10, so it's available across the entire supported range — no version gating needed.

And the CLI can't decide either. codelinks has a CLI (codelinks analyse → JSON) that reads markers with no Sphinx build present — we don't interpret the RST at that stage at all, and don't know whether it will ever render in a Sphinx or ubCode scope. So "is this RST" can't be decided at extraction.

My position — this makes it work cleanly: on the Sphinx level, reject marked-RST content when src-trace is not in an RST document: raise a warning and skip extraction. Then "RST-ness" is guaranteed by construction — in an RST host it parses as RST, and we never silently reinterpret it as Markdown. The CLI stays format-agnostic (raw text out); the Sphinx layer enforces the contract.

@ubmarco All three points addressed in a2c4839:

• Replaced nested_parse_with_titles with nested_parse_to_nodes(…, allow_section_headings=False) from sphinx.util.parsing. No version gating needed — nested_parse_to_nodes is available across the entire supported sphinx>=7.4,<10 range.
• Added an early-return guard in _render_marked_rst: checks self.env.doc2path(self.env.docname) suffix and emits a logger.warning + returns [] for non-RST hosts, preventing silent Markdown misparse in MyST documents.

[PhilipPartsch]

Interesting. I believe we have to give the task to ensure we only add rst content from comments to rst files to the user.

Why?

Reason:
The user could define two markers in one comment:

c file:

/*
@rst
.. impl:: implement dummy function 1
   :id: IMPL_MRST_BASIC_1

   Body paragraph inside the marked RST block.
@endrst

@md
:::{impl} implement dummy function 1
:id: IMPL_MMD_BASIC_1

Body paragraph inside the marked RST block.
:::
@endmd
*/

with multi project config
toml:

[codelinks.projects.my_project_rst.analyse]
get_rst = true
[codelinks.projects.my_project_rst.analyse.marked_rst]
start_sequence = "@rst"
end_sequence = "@endrst"

[codelinks.projects.my_project_md.analyse]
get_rst = true
[codelinks.projects.my_project_md.analyse.marked_rst]
start_sequence = "@md"
end_sequence = "@endmd"

and fetch in his rst files the rst section

rst file:

.. src-trace::
   :project: my_project_rst

and in his md files the md section

md file:

:::{src-trace} 
:project: my_project_md
:::

@ubmarco: maybe we should rename the config parameter get_rst to get_content_from_comment. What do you think?

OR:

We could even think, to set the parsing language by configuration:

[codelinks.projects.my_project_xyz.analyse]
get_content_from_comment = true
set_text_style = <rst | myst md>
[codelinks.projects.my_project_xyz.analyse.marked_rst]
start_sequence = "@xyz"
end_sequence = "@endxyz"

With this, we may could even support to parse rst comments to md files or vice versa, as we could change the parser. Not sure about, may have to ask @chrisjsewell?

[ubmarco]

@cpolzer thanks for the fixing round
@PhilipPartsch valid point
@chrisjsewell will tackle the topic next week. We are planning to migrate codelinks into sphinx-needs, to test everything in one unit, and integrate better into internal sphinx-needs workings. Please be a bit patient.