[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-08-12

[github-actions[bot]]

Summary

• Date of test: 2026-08-12
• Pages visited:
  • /gh-aw/ (Home)
  • /gh-aw/setup/quick-start/ (Quick Start Guide)
  • /gh-aw/setup/cli/ (CLI Commands)
• Overall impression: As a brand-new user, the home page and Quick Start are genuinely well organized — prerequisites, numbered steps, and copy-paste commands are all present. My main friction point is the sheer size of the left sidebar, which can feel overwhelming before you've even started.

Critical Issues Found

None encountered that would block a first-time user from completing the Quick Start guide. Prerequisites, install command, add-wizard command, and compile command are all present, accurate, and in a logical order.

Confusing Areas

1. Overwhelming sidebar navigation (all 3 pages) — The left nav lists dozens of reference pages (Reference section alone has 40+ entries: Frontmatter, Sandbox, Safe Outputs, Threat Detection, etc.) visible immediately on first visit, before a beginner has even run their first workflow. This creates an impression of high complexity right away. A collapsed-by-default "Reference" and "Design Patterns" section (expand-on-click) would keep the sidebar approachable for newcomers.
   📎 quickstart-top.png

2. Jargon introduced without immediate definition — Terms like "frontmatter" and "lock file" (.lock.yml) are used in Step 2/Step 4 before being fully explained; a first-time reader has to click through to the Frontmatter reference page or the "Lock File" anchor to understand what's meant. A one-line inline definition on first use (it's partially done for "lock file" via a Note callout, but "frontmatter" only gets a link) would help.

3. Engine authentication step is dense — Step 2's authentication instructions (Copilot/Claude/Codex/Gemini/Pi tabs) pack a lot of conditional info (PAT creation, copilot-requests: write, env vars) into one step. For a Copilot-only new user this is mostly skippable, but it's not obvious at a glance that most of it can be ignored if you already have Copilot access with org billing.

What Worked Well

• Home page clearly states what the tool does in one sentence and has a prominent "Quick Start with CLI" call-to-action button plus a "Quick Start" top-nav link — easy to find.
• Quick Start prerequisites list is concrete and actionable (AI account, repo write access, GitHub Actions enabled, gh CLI version + login check) with direct links to install/configure each.
• Step numbering (1-4) with copy-to-clipboard code blocks makes the happy path very easy to follow without needing to understand Git/Actions internals first.
• Tip/Caution/Note callouts proactively address common issues (auth failures, what --engine claude does, why the .lock.yml file should be committed) — this anticipates beginner confusion well.
• CLI Commands page leads with a concise "Most Common Commands" table (init, doctor, add-wizard, add, new, compile, list, run, status, logs, audit) with a "When to use" column, plus an explicit tip: "New to gh aw? You can stop here for day-one usage." This is a great pattern for reducing overwhelm — the full command reference is available but the newcomer path is triaged first.

Recommendations

Quick wins:

• Collapse the "Reference" and "Design Patterns" sidebar sections by default so newcomers see a shorter, more approachable list on first visit.
• Add a one-line inline definition for "frontmatter" the first time it's used in the Quick Start guide (similar to how "lock file" already gets a Note).
• Consider a "if you just want Copilot with org billing, skip to X" shortcut note above the auth tabs in Step 2 for the common case.

Longer-term:

• Consider applying the CLI Commands page's "Most Common X" triage pattern (a short table + "stop here for day one" tip) to other large reference-heavy pages that beginners land on early.
• A brief glossary tooltip/hover for jargon terms (frontmatter, lock file, safe outputs, engine) throughout docs would reduce the need to context-switch to the Glossary page.

Screenshots

📎 [home.png] — Home page, clear CTA and nav — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/accbf7165877c9d3d4e6498d75c0a8b596dfad22efd0ff7b7852a7cc55be6fb9.png?raw=true

📎 [quickstart-top.png] — Quick Start page top, showing large sidebar — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/54e35214977e5f752ddcc7987a36b77433dc410042aacdc271d5298ce33be6d4.png?raw=true

📎 [quickstart-full.png] — Full Quick Start page (all steps) — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/9e7648bbc0eeb56d7142363a0173ae92ab3bbd710a81c378674f9f7349fcdfb5.png?raw=true

📎 [cli-commands.png] — CLI Commands page, "Most Common Commands" table — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d36276e052319bea3d66d81470ccaed7f6270c51b85d347f5ea4c73fd59a8895.png?raw=true

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · auto · 42.9 AIC · ⌖ 4.21 AIC · ⊞ 9.7K · ◷

• expires on Aug 12, 2026, 8:26 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #52426.