Skip to content

Score how completely both readers decode the pinned corpus - #169

Draft
Stephen316 wants to merge 4 commits into
feat/pr94-automation-only-20261117from
codex/read-scoreboard
Draft

Stephen316 wants to merge 4 commits into
feat/pr94-automation-only-20261117from
codex/read-scoreboard

Conversation

@Stephen316

@Stephen316 Stephen316 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Change

Add re/automation/ReadScoreboard.py. It decodes every pinned native file in re/automation/Corpus.json (714 Fusion, 58 SOLIDWORKS parts, 6 SOLIDWORKS assemblies) and reports how completely each reader understands them. The goal is a measurable path to a complete read for both applications before writer work.

For each file the scoreboard records:

  • the outcome: strict read, partial read only, rejected or reader error;
  • the strict refusal reason, if any;
  • every unresolved item and its reason, attributed to the feature, sketch or document area at its path. SOLIDWORKS steps use their stored native type. Fusion native steps use the class name from re/Fusion/Study/fusion-class-catalog.json;
  • fully_read: strict mode accepts the file and no unresolved item remains.

Scoreboard.json and Scoreboard.md give totals per application, the areas that block the most files, unresolved reasons, strict refusals and Fusion changes since FusionOutcomes.json.

ReaderEvidence.reader_evidence gains an optional keyword limit. The default stays 200,000 entries, so FusionReader.py, InferenceRunner.py, NeutralReaderRunner.py and SolidworksReader.py are unchanged. The scoreboard passes 10,000,000, because the four pinned assemblies need between 0.9 and 5.8 million entries.

This stacks on #158.

Validation

  • tests/Research/Census/ReadScoreboardTests.py runs the command on pinned Fusion strict, Fusion partial and SOLIDWORKS files, including an accented path in NFD form, and checks the refusal of a reused output directory and of unpinned paths. Both cases passed locally.
  • Scored alone, ENSAMBLE 3_IZQUIERDO.SLDASM now decodes as a partial read instead of exceeding the evidence limit.
  • Measured evidence sizes: ENSAMBLE 3_IZQUIERDO 896,894; Cylinder Head Assy 1 958,093; Turbo Assy KF 1,230,790; Engine Assy KF EDIT 2 5,828,609 entries (19.3 s, 2.7 GB peak).

Not done yet

  • The full 778-file run has not completed. Its numbers will go in the change note.
  • The change entry, Documentation/Changes/Research/Automation/ReadScoreboard.md, will get the full-run totals once that run is recorded.
  • Byte ownership is not measured: the reader evidence reports byte_accounting: unavailable.
  • Hosted CI is pending.

Contributions are provided under the MIT License.


Devin Review

@github-actions

Copy link
Copy Markdown
Contributor
Diff intelligence · initial analysis · 388 changed lines · 90 reviewable

PR #169 · 734ad41d32

Total PR diff Reviewable Excluded from review
388 90 76.8%

+385 additions · −3 deletions · 3 files · 0 binary files

Reverse Engineering: 76.8%Tests: 23.2%

Category Additions Deletions Lines Share Files Policy
Reverse Engineering +295 −3 298 76.8% 2 excluded
Tests +90 −0 90 23.2% 1 review

The bar follows the table order. Shares use additions + deletions and reconcile to 100% when text changes exist. Binary files have no fabricated line counts.

Review focus

90 reviewable lines; 298 lines excluded by configured policy. This metric is not a guarantee of low risk.

Classification: 3 files matched deterministic rules or Git metadata; 0 unclassified files remain reviewable. Rule matches express configuration confidence, not semantic or security certainty.

Largest changed files
File Category Status Additions Deletions
re/automation/ReadScoreboard.py Reverse Engineering A +290 −0
tests/Research/ReadScoreboardTests.py Tests A +90 −0
re/automation/ReaderEvidence.py Reverse Engineering M +5 −3
Unclassified, renamed, deleted, binary and metadata-only files

Base a6062eae8e · merge base a6062eae8e
Classification b546a07fa1fd5092 · trusted implementation b79fb8ea5b
Complete per-file JSON and Markdown · event 44636784007:1

@github-actions github-actions Bot added the Tests label Nov 19, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

@Stephen316 Stephen316 changed the title WIP - Score how completely both readers decode the pinned corpus Score how completely both readers decode the pinned corpus Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor
Diff intelligence · update #2 · 437 changed lines · 93 reviewable

PR #169 · 47c1794e31

Total PR diff Reviewable Excluded from review
437 93 78.7%

+434 additions · −3 deletions · 4 files · 0 binary files

Documentation: 10.5%Reverse Engineering: 68.2%Tests: 21.3%

Category Additions Deletions Lines Share Files Policy
Documentation +46 −0 46 10.5% 1 excluded
Reverse Engineering +295 −3 298 68.2% 2 excluded
Tests +93 −0 93 21.3% 1 review

The bar follows the table order. Shares use additions + deletions and reconcile to 100% when text changes exist. Binary files have no fabricated line counts.

Review focus

93 reviewable lines; 344 lines excluded by configured policy. This metric is not a guarantee of low risk.

Classification: 4 files matched deterministic rules or Git metadata; 0 unclassified files remain reviewable. Rule matches express configuration confidence, not semantic or security certainty.

Composition changes

Previous 734ad41d32 → current 47c1794e31

Net snapshot change: +49 total lines, +3 reviewable lines, +1.9 percentage points excluded.

Category Previous Current Delta
Documentation 0 46 +46
Reverse Engineering 298 298 +0
Tests 90 93 +3

What changed in this push

Actual head-to-head diff: +79 / −30 across 2 files (0 binary).

  • Documentation: 1 files, +46 / −0
  • Tests: 1 files, +33 / −30

Net PR snapshot changes are not the lines edited during this push.

Largest changed files
File Category Status Additions Deletions
re/automation/ReadScoreboard.py Reverse Engineering A +290 −0
tests/Research/Census/ReadScoreboardTests.py Tests A +93 −0
Documentation/Changes/Research/Automation/ReadScoreboard.md Documentation A +46 −0
re/automation/ReaderEvidence.py Reverse Engineering M +5 −3
Unclassified, renamed, deleted, binary and metadata-only files
Reviewable size over recent pushes

Base a6062eae8e · merge base a6062eae8e
Classification b546a07fa1fd5092 · trusted implementation b79fb8ea5b
Complete per-file JSON and Markdown · event 44637710106:1

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 19, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Custom reviewers

  • 📖 Kit Acceptance: no findings · View

Devin Review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Additions documentation Improvements or additions to documentation LOW Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant