Skip to content

merge-graphs does not namespace the community node attribute, so communities collide across repos and the aggregated HTML view fuses unrelated ones into a single meta-node #3014

Closed

Description

@temsecanvil

merge-graphs prefixes node ids with the repo tag, rewrites edge _src/_tgt, prefixes hyperedge ids and remaps their members, and stamps repo on every node — but the per-node community attribute is carried across unchanged. Each input graph numbers its communities from 0, so the merged graph inherits N overlapping id spaces.

That is invisible in graph.json itself, but the aggregated community view (taken for any merged graph above the 5000-node viz limit) groups by the bare community id, so communities from different repos that happen to share an integer are drawn as one meta-node.

Reproduction

Merging three unrelated repos (11707 + 2057 + 1892 nodes; 754 + 203 + 172 communities):

$ graphify merge-graphs A/graphify-out/graph.json B/graphify-out/graph.json C/graphify-out/graph.json --out merged/graphify-out/graph.json
Merged 3 graphs -> 15656 nodes, 25936 edges

distinct community ids: 754        # expected 1129 = 754 + 203 + 172
community ids shared by >1 repo: 203

754 is exactly the largest input's community count, which is what makes it read as plausible rather than wrong.

Community 0 in the merged graph holds 227 nodes drawn from all three repos:

community 0 -> A: 114 nodes  "Preflight Check Suite"
               B:  49 nodes  "Canary Disk Budget"
               C:  64 nodes  "Heading Declaration Gate Tests"

Three unrelated communities, one meta-node:

$ graphify export html --graph merged/graphify-out/graph.json
Graph has 15656 nodes (above 5000 limit). Building aggregated community view...
graph.html written (aggregated: 754 community nodes, 1577 cross-community edges)

After namespacing (repo, community) locally, the same graph exports 1129 community nodes.

Mechanism

build.py::prefix_graph_for_global disambiguates everything else per repo but never touches community. Its own comment credits PR #1691's diagnosis for the hyperedge half:

relabel = {n: f"{repo_tag}::{n}" for n in G.nodes}
H = nx.relabel_nodes(G, relabel, copy=True)
for node, data in H.nodes(data=True):
    data["repo"] = repo_tag
    data.setdefault("local_id", node.split("::", 1)[1])
# ... edges (_src/_tgt), hyperedge ids and members all remapped ...
# 'community' is not remapped

The consumer side then groups on the bare id — cli.py, the export html path that reconstructs communities from node attributes:

reconstructed: dict[int, list[str]] = {}
for node_id, data in G.nodes(data=True):
    cid_raw = data.get("community")
    ...
    reconstructed.setdefault(cid, []).append(str(node_id))

and exporters/html.py emits one meta-node per key:

node_to_community = {nid: cid for cid, members in communities.items() for nid in members}
for cid, members in communities.items():
    meta.add_node(str(cid), label=(community_labels or {}).get(cid, f"Community {cid}"))

merge-graphs writes no .graphify_analysis.json, so a merged graph always takes that reconstruction path rather than a sidecar.

Prior art

PR #1691 ("fix(merge): preserve repository-local identities") proposed keeping "existing community partitions distinct for each repository" and noted "Community IDs overlap". It was closed unmerged in favour of the fix shipped for #2484, which covers the hyperedge half of the same PR; the community half does not appear in the shipped function. #2134 separately describes merge-graphs and the global graph as unioning "repositories side by side", which matches what I see.

Workaround, and why I still think this is worth fixing

graphify cluster-only <merged-dir> --graph <merged graph.json> --no-label re-partitions the merged graph and yields unique ids (1044 communities here, none spanning a repo). It discards the curated per-repo community names, so those have to be regenerated.

So there is a one-command fix, and you could reasonably say the raw merge is simply un-analysed and re-clustering was the missing step. Two things make that a harder sell from the user side:

  • extract tells you to do it and merge-graphs does not. extract prints a "next: run graphify cluster-only ..." line; merge-graphs prints only the node/edge counts and the output path, and references/github-and-merge.md goes straight from merge-graphs to querying the merged graph with no re-cluster step.
  • Nothing detects the bad state. export html silently reconstructs communities from the node attribute and renders the fused view; export wiki does have a guard for untrustworthy community data, but it only fires when communities is empty, and here it is populated and wrong. So the failure is silent in every path.

Even just prefixing the ids at merge time, or printing the same "run cluster-only next" hint that extract prints, would close it.

Suggested fix

Namespace community in prefix_graph_for_global alongside the hyperedge ids that are already prefixed there — either f"{repo_tag}::{cid}" or an offset per input — and key any combined community_labels the same way. Keeping the pre-merge value in something like local_community would preserve the per-repo partition for callers that want it.

Environment

graphify 0.9.49 (pipx), Python 3.13, Debian 13. Also reproduced on 0.9.48.

Activity

  1. temsecanvil commented on Aug 24, 2026

    @temsecanvil
    Author

    Independent reproduction on a second machine, with different numbers, which I think is worth adding because the failure mode is a plausible-looking number rather than an obvious break.

    A colleague ran the same three repos through the same pipeline on Windows, with different chunk boundaries and a separately-built extraction (so the per-repo graphs are not copies of mine):

    my run second run
    tempered-ansible 11,707 nodes / 754 communities 11,711 / 856
    temsecdocs 2,057 / 203 2,639 / 207
    temsectools 1,892 / 172 2,017 / 218
    merged, distinct community ids 754 (true total 1,129) 856 (true total 1,281)
    ids spanning >1 repo 203 218

    The point of including both: in each case the merged graph inherits exactly the largest input's community count — 754 and 856 are each that run's own tempered-ansible community count. So the symptom is not a crash or an obviously wrong number, it is a number that looks entirely reasonable for a merged graph and silently is not. Confirmed end to end on the second run too: the aggregated HTML draws 1,281 community nodes after namespacing (repo, community), and 856 unfixed.

    The second reader also independently traced it to the same place — prefix_graph_for_global rewrites node ids, repo, local_id, edge _src/_tgt and, since #2484, hyperedge ids and member lists, and never touches community.

    Worth noting the precedent that sits inside that same function: this exact collision class was already hit for hyperedges and fixed by namespacing them (#2484, following #1691 "preserve repository-local identities"). community looks like the one attribute the same reasoning was never extended to, which is also why the fix is likely a couple of lines in a place that already does this for its neighbours.

    One small addition to the workaround note above: to_html(...) raises ValueError: too large for HTML viz when node_limit is left None, so node_limit=5000 is what actually selects the aggregated community view — relevant to anyone trying to reproduce the fused render directly rather than through the CLI.

  2. safishamsi commented on Aug 25, 2026

    @safishamsi
    Member
  3. added a commit that references this issue on Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions