Expose the production robots.txt contract

[alexeygrigorev]

Raw intake

The authoritative SEO specifications require production to expose a versioned robots.txt crawl contract alongside the production sitemap. Current core.views.robots() only serves the development Disallow: / body and raises 404 whenever NOINDEX=False; the production-shaped test explicitly codifies that 404.

Keep development non-indexable behavior unchanged, but make production return the approved crawl/robots body with the correct content type, method/cache/header behavior, and tests. Do not alter sitemap routes, canonical URLs, deployment host policy, or use robots as authorization.

[alexeygrigorev]

Grooming Complete

The issue is implementation-ready as a P0 production SEO/compatibility fix. The current
development deployment is intentionally non-indexable; production has no application endpoint
for the accepted robots.txt contract because core.views.robots() returns the development body
only when NOINDEX=True and otherwise raises 404.

Normative authority and exact contract

• Spec 02 — URL, link, and SEO compatibility is authoritative for the public route/cache/SEO contract. It classifies robots.txt and sitemap output as explicit discovery artifacts, requires production crawl/sitemap output, and requires development to remain disallowed and non-indexable.
• Spec 07 — Security, privacy, accessibility, and operations makes robots.txt a crawl preference, never an authorization control, and requires private/no-store behavior to remain fail-closed for credential-bearing traffic.
• Spec 10 — Verification strategy requires production robots/sitemap/indexing parity and GET/HEAD/cache/error coverage.
• The checked compatibility baseline records /robots.txt as a source-backed 200 text contract in generated-path-baseline.jsonl and public-contracts.jsonl. The pinned legacy source revision is ee43d3fa0929faf691178d79f19528e6f15a83e5; its production body is the preserved disallow policy. The later owner decision in ab11119 retires /podwiki/ and makes /wiki/ the sole Wiki family, so the robots sitemap hint must not advertise the retired /podwiki/sitemap.xml path.

The production body is UTF-8, byte-stable, ends with one newline, and is exactly:

User-agent: *
Disallow: /admin/
Disallow: /_site/
Disallow: /drafts/
Disallow: /config/
Disallow: /scripts/
Disallow: /styles/
Sitemap: https://datatalks.club/sitemap.xml
Sitemap: https://datatalks.club/sitemaps/wiki.xml

The first six directives preserve the pinned production source policy. The first sitemap is the
canonical ten-section index already emitted by content.public_views.production_sitemap(); the
second is the current canonical Wiki section route from
content.sitemap_contract.EXPECTED_SITEMAP_LOCATIONS. No /podwiki/ URL may appear in the new
body. This issue does not change the sitemap index or section contents/routes.

Environment branches are exact:

• Production (NOINDEX=False): anonymous GET /robots.txt returns 200, the body above,
  Content-Type: text/plain; charset=utf-8, browser Cache-Control: max-age=0, must-revalidate,
  and no X-Robots-Tag header. Anonymous HEAD has the same status and headers and an empty
  body. The current all-zero-TTL deployment baseline remains in force; this issue does not add
  positive s-maxage caching.
• Development/preview (NOINDEX=True, including web.dtcdev.click): preserve the current
  200 UTF-8 text/plain body User-agent: *\nDisallow: /\n, empty HEAD body, exact
  X-Robots-Tag: noindex, nofollow, and no Sitemap: directive. The development sitemap may
  remain the production-origin index only for compatibility inspection and stays covered by
  noindex plus disallow-all robots behavior.
• Unsafe methods and errors: POST, PUT, PATCH, and DELETE (and any other non-GET/HEAD
  method) return 405 with Allow: GET, HEAD, no response body contract, and cache directives
  containing no-store and max-age=0 with no public or positive s-maxage. They must not
  become an authorization mechanism or disclose implementation details.
• Credential-bearing requests: an authenticated/session/credential-shaped request never gets
  a shared-cacheable response; the existing private/no-store middleware contract wins. Robots
  remains a crawl hint and never grants or denies application access.

Scope

• Add a production branch to the robots view with the exact body/content type above and the
  browser cache directive for the current zero-TTL baseline.
• Preserve the development branch and its edge/application noindex, nofollow behavior byte-for-byte.
• Make method handling explicit enough to preserve exact GET/HEAD parity and fail-closed unsafe
  responses.
• Replace the production-shaped 404 assertion with the exact production body/header/cache
  assertions; retain the existing development tests.
• Add focused repository and browser-request coverage for both settings branches, body bytes,
  content type, status, Allow, Cache-Control, X-Robots-Tag, and absence of the retired
  /podwiki/ sitemap hint. Verify that the existing ten-section sitemap index and section routes
  remain unchanged.

Explicit non-goals

• Do not change /sitemap.xml, /sitemaps/<section>.xml, sitemap XML contents, canonical URLs,
  public content, route aliases, or the route inventory beyond the robots text itself.
• Do not add /podwiki/ redirects or resurrect any retired Wiki route; do not redesign the /wiki/
  route family.
• Do not add Disallow rules for new application paths, invent a private-route crawler policy, or
  use robots as authentication/authorization/rate limiting.
• Do not enable positive CloudFront caching, change cache keys/TTL policies, add WAF rules, alter
  DNS/Route 53/ALB/host policy, or submit a sitemap to Search Console. Positive caching belongs to
  #109, and deployment infrastructure is
  outside this bounded origin contract.
• Do not alter analytics, content projections, legal copy, credentials, secrets, production data,
  or external provider integrations. No live production probe or deployment is performed during
  engineering/tester work without the separately authorized environment.

Dependencies and ownership

• The accepted development safety behavior from #36
  is a compatibility prerequisite and must remain green; this issue changes only its deliberate
  production-shaped negative branch.
• #109 owns the future generated route-cache
  registry and bounded positive CloudFront matrix. It is a follow-up integration consumer, not a
  blocker for the application branch while the current baseline is zero TTL; when Cache anonymous public traffic safely at CloudFront #109 lands,
  /robots.txt must retain this body/browser contract and receive only its approved public
  discovery cache class.
• #160 currently blocks readiness/service
  count and any exact live deployment proof. It is not a blocker for local implementation or
  isolated settings/browser tests; production smoke remains a post-deployment verification gate.
• Preserve trailing-slash Docs and FAQ root routes #163 is already merged and its /wiki/ decision is the authority for replacing the stale
  /podwiki/sitemap.xml hint. No owner/content decision is open for this issue.
• The engineer owns core.views, focused core SEO tests, and any narrowly necessary deployment
  contract fixture changes. The tester independently reruns the matrix and records request/header
  evidence under .tmp/ without credentials or production data. The PM accepts only after that
  independent verification; on-call observes the exact post-merge CI run.

Acceptance criteria

Production application contract

• With NOINDEX=False, anonymous GET /robots.txt is 200, has exactly the body above and
  final newline, Content-Type: text/plain; charset=utf-8, browser
  Cache-Control: max-age=0, must-revalidate, no X-Robots-Tag, and no Location/canonical
  HTML output.
• With NOINDEX=False, anonymous HEAD /robots.txt is 200, has the same content type,
  cache and robots headers as GET, and has an empty transfer body.
• Production body contains exactly the six preserved Disallow directives and the two
  canonical production Sitemap URLs; it contains no /podwiki/, development host, query,
  fragment, or secret-bearing value.
• Production POST, PUT, PATCH, DELETE, and other unsafe methods return 405, exact
  Allow: GET, HEAD, and fail-closed no-store/max-age=0 cache directives. No unsafe method
  mutates state, performs network work, or emits a redirect.
• Production requests carrying a session, Authorization, CSRF, preview/management token, or
  unknown credential-shaped cookie cannot receive or populate a shared-cacheable robots response;
  the existing private/no-store response policy remains authoritative.

Development and sitemap compatibility

• With NOINDEX=True, existing development GET/HEAD robots behavior remains byte-exact:
  200, UTF-8 text/plain, disallow-all body, empty HEAD body, exact
  X-Robots-Tag: noindex, nofollow, and no Sitemap: directive.
• Existing development /sitemap.xml and all ten section sitemap locations remain unchanged;
  its production-origin URLs remain protected by development noindex/disallow-all behavior.
• No canonical, Open Graph, route alias, public path, projection, content, analytics, or
  deployment-host behavior changes as a consequence of this issue.

Automated/repository evidence

• Focused Django tests cover both NOINDEX branches, GET/HEAD parity, exact bytes/content
  type, cache directives, robots headers, unsafe-method denial, credentials/private cache policy,
  and the two sitemap hints. Existing core/tests/test_development_seo.py production-shaped
  404 coverage is replaced by this positive production contract; development assertions remain.
• Core Playwright/request tests exercise production-shaped and development settings without
  contacting live production, assert the same body/status/header matrix, and verify no accidental
  /podwiki/ or web.dtcdev.click sitemap URL. Screenshots are not required for a text-only
  endpoint; any captured artifacts stay below .tmp/screenshots/ and contain no sensitive data.
• content/tests/test_sitemap_contract.py and existing route/SEO tests remain green, proving
  the accepted ten-section sitemap index and canonical section locations are untouched.
• Repository checks run through uv/Make targets: focused Django/core browser checks,
  make check-seo, make check-links, make django-check, and the applicable lint/format/type/
  migration checks. git diff --check is clean and no source/compatibility artifact is regenerated
  without an explicit contract reason.

Browser/deployed scenarios

• In a local production-settings browser/request session at desktop and mobile viewport
  profiles, fetch /robots.txt and /sitemap.xml; verify the exact production body, headers,
  no development X-Robots-Tag, valid unchanged sitemap index, and no redirect/debug response.
• In a local development-settings browser/request session, fetch /robots.txt, /sitemap.xml,
  and a known section sitemap; verify disallow-all/noindex behavior, production-origin sitemap
  locations, no analytics/network side effects, and empty HEAD bodies.
• Exercise unsafe POST /robots.txt and a credential-bearing request in the local matrix;
  verify 405/Allow, no-store behavior, no body reflection, and no origin mutation/network
  side effect. A future exact-SHA deployment smoke must repeat the same GET/HEAD/status/content
  type/cache/robots-header checks against the production host after Restore development database readiness for web.dtcdev.click #160 permits a safe release.

Ready for software engineering. No source, deployment, AWS, secret, production-data, commit, push,
or pull-request action was performed during grooming.

[alexeygrigorev]

Software Engineer Report

Issue #164 is implemented in the dedicated, uncommitted worktree
/home/alexey/git/dtc-website/.tmp/website-164-engineer.

Candidate provenance

• Base SHA: 754bc24cb0510c9894a07686fae925935dcaf754
• Head SHA: 754bc24cb0510c9894a07686fae925935dcaf754 (the candidate is an uncommitted worktree
  overlay on the exact pinned origin/main tree)
• Verification source mode: worktree
• Direct graph node: shared.core
• Profile/reason: full / render_impact (basename:views.py)
• Graph SHA-256: 4127118b4467a75b40b8dbadc4050b5b2dc3762b292aca93cd9309a4593bca19
• Policy version: 4
• Risk flags: shared_runtime, test_infrastructure
• Changed paths only:
  • core/views.py — SHA-256 3a9f5b165179b0581d960fd1dc1c7e404eed063214b8d3d1d3c7980734cd74c7
  • core/tests/test_development_seo.py — SHA-256 43343b8a487243194dd97aa08004e992af38af0e4949519eaacbb11a6407d5d1

No deployment, AWS, route, sitemap, compatibility artifact, secret, production-data, commit, or
push action was performed.

Implementation

• core.views.robots() now rejects every method other than GET/HEAD with 405, exact
  Allow: GET, HEAD, and Cache-Control: no-store, max-age=0.
• With NOINDEX=False, GET and HEAD return the byte-stable UTF-8 production policy with one
  final newline, text/plain; charset=utf-8, and browser Cache-Control: max-age=0, must-revalidate; the response does not add X-Robots-Tag.
• The production body preserves the six pinned Disallow directives and advertises only
  https://datatalks.club/sitemap.xml and the current canonical
  https://datatalks.club/sitemaps/wiki.xml section. It contains no retired /podwiki/ or
  development-host hint.
• With NOINDEX=True, the existing User-agent: *\nDisallow: /\n body and middleware-owned
  X-Robots-Tag: noindex, nofollow behavior remain unchanged. Existing sitemap generation and
  routes are untouched.
• Credential-shaped requests continue through ResponsePolicyMiddleware, which converts the
  public browser directive to the established private/no-store policy; robots remains a crawl
  preference and never authorization.

Focused evidence

• DTC_TEST_RUN_ID=issue-164-robots-2 uv run --frozen python manage.py test core.tests.test_development_seo.DevelopmentRobotsAndSitemapTests --noinput: 3 tests passed.
  This covers production GET/HEAD bytes, content type/cache/robots headers, all unsafe methods,
  credential-shaped requests, absence of /podwiki//development-host hints, and unchanged sitemap
  output.
• DJANGO_ALLOW_ASYNC_UNSAFE=true DTC_TEST_RUN_ID=issue-164-browser uv run --frozen pytest playwright_tests/test_development_seo.py -k preview_token_and_response_matrix_are_safe -m core -v: 1 existing development browser/request test passed, confirming disallow-all/noindex and
  sitemap behavior without contacting live production.
• make check-seo: 76 passed.
• make check-links: 7 passed.
• make django-check: OpenAPI current, management parity current, Django check clean.
• make migrations-check: no changes detected.
• make lint, make format-check, and make typecheck: all passed (372 files type-checked).
• git diff --check: clean.

Versioned verification plan and report

• Plan: .tmp/verification/issue-164-v2/verification-plan.json
• Plan SHA-256: d97a6af5c9aaa395e609701a27e6ba39d8153e61b5a6e5f87ad9aa3ca86dd14a
• Verification state SHA-256: 77581f2c748181dda9087ca8ecac338b467f8cb554f3eb56ae7be28fe9fefc40
• Engineer report: .tmp/verification/issue-164-v2/verification-report.json
• Engineer report SHA-256: cf9d5f9833e7f35a3141ebb301e642fc556a90f2e4babb6ae4ed37449b0e8a8f
• Plan/report validation completed through make verification-report-check.

All rerun components succeeded:

• Django: 2,108 assertions, 2,100 passed, 8 expected skips; evidence ID
  85804fb3e61f41fee8c8631a5d5d68cb027f4f44ddfee8baab4299b6d0143ed7; output SHA-256
  3295b13f6096a335b74defe81abfde1211d8552d91a81873a40c955ce45c8ed1.
• Playwright core/full: 181 passed, 0 failed; evidence ID
  485b63321f4f2c8b725f599e5c0cbd43fdf2766f462b460e04cf57e2e35067fe; output SHA-256
  721fc81c627c8a52923f0ce60f5f33f36a8917f3341b0edecc2b320881738d94.
• Quality contract: 531 passed; evidence ID
  141ee55255818ef2eef95fb3ddd03d80f3ac9e9239ab3d306d422f7e5a008372.
• Evidence validation: 531 passed; evidence ID
  9e98d9942820f56389a1f7c4ac6e2c0b6021b7f4df86a5b8fd92cce7fd550fb3.
• Container: 12 assertions passed; evidence ID
  ff2c9b81036ea89f41fcce4b4f752fd30bc33634635b98d52615d0bcd566b16e.
• Selector/control plane: success; evidence ID
  1b2243be7694483932fbf73be86f6aaab9a14c2e1282dc323fc7bf9c877ddb64.
• compatibility and content_invariants are not_applicable because they are outside this
  source-impact closure; the full Django command still included its normal compatibility subset.

Screenshot disposition and handoff state

The planner marks screenshots as skipped: independent_tester_required because the changed
core/views.py basename triggers the repository render-impact policy. It derives eight desktop/
mobile captures for the home, not-found, courses, and Studio sign-in routes. The robots endpoint is
text-only and no new page screenshot is claimed here; existing Playwright-produced images are not
independent tester evidence. The verification verdict is therefore pending_independent_tester,
not an engineer pass of the screenshot gate.

The candidate is frozen uncommitted for the independent tester. The tester must recompute the plan
from this exact base/head, validate the retained evidence, independently run the changed-contract
matrix, and resolve the graph screenshot gate before PM acceptance. Live production smoke remains
outside this handoff until the separately authorized deployment/readiness dependency is available.

[alexeygrigorev]

Software Engineer Report — #164 cache-boundary supersession

Status: candidate frozen for independent tester review. The candidate is uncommitted and
unpushed in the isolated worktree below. Product acceptance and the tester gate remain open.

Why this supersedes the prior handoff

The independent tester's production-shaped request matrix identified a release-blocking cache
boundary gap: /robots.txt GET and HEAD responses with only a CSRF cookie, an unknown cookie, an
X-Preview-Token, or an X-Management-Token still carried anonymous cache directives. The tester
artifact used for that finding is synthetic and its SHA-256 is
1134af588c33c631ebf8aa7dc2351981185bcf00b91ce0e6752f9f21afe74442.

The earlier handoff comment
(issuecomment-5299195377)
and its v2 verification artifacts are INVALIDATED/SUPERSEDED and must not be reused:

• .tmp/verification/issue-164-v2/verification-plan.json, SHA-256
  d97a6af5c9aaa395e609701a27e6ba39d8153e61b5a6e5f87ad9aa3ca86dd14a;
• .tmp/verification/issue-164-v2/verification-report.json, SHA-256
  cf9d5f9833e7f35a3141ebb301e642fc556a90f2e4babb6ae4ed37449b0e8a8f.

The intermediate v3 run is also superseded: its first fail-closed classifier treated the
documented browser_timezone preference as unknown and the full browser gate exposed that
behavioral regression. v4 below is the final candidate state.

Candidate identity and ownership

• Repository: DataTalksClub/website.
• Exact base and head: 754bc24cb0510c9894a07686fae925935dcaf754 (pinned origin/main; the
  candidate changes are uncommitted worktree edits).
• Isolated worktree: /home/alexey/git/dtc-website/.tmp/website-164-engineer.
• No commit, push, deployment, or production mutation was performed.

Changed files and final SHA-256 values:

• core/middleware.py — a995a0ee09796a14119be90b9c406503f5079d7ad67c0df24ee2d6683cf35691;
• core/tests/test_development_seo.py — eeb688609268924c8340e1386934d3ba5109c89fe33bb770c1ee094109b033a8;
• core/views.py — 3a9f5b165179b0581d960fd1dc1c7e404eed063214b8d3d1d3c7980734cd74c7.

Diagnosis and focused fix

ResponsePolicyMiddleware previously classified only an authenticated Django user, an
Authorization header, or the session cookie. Early responses can bypass user resolution, and
viewer credentials such as CSRF, preview, management, or unknown credential-shaped cookies do not
necessarily establish a user. The shared cache policy therefore failed closed too late.

The candidate now classifies credential-bearing requests before applying the response policy by
checking:

• authenticated users and Authorization;
• X-CSRFToken, X-Preview-Token, and X-Management-Token;
• configured session and CSRF cookie names; and
• every cookie except the two documented non-credential preferences, browser_timezone and
  dtc_analytics_consent.

Credential-bearing responses are forced through the existing overwrite helper to
private, no-store, max-age=0, with public and positive shared-cache directives removed. The
known preferences remain anonymous-safe so existing public responses do not regress.

robots.txt keeps the issue's production contract: exact production body, text content type,
GET/HEAD only, max-age=0, must-revalidate for anonymous production requests, private/no-store
for credentials, and 405 Allow: GET, HEAD with no-store, max-age=0 for unsafe methods.
Development still serves the exact disallow-all body with noindex; sitemap behavior is unchanged.

Focused verification

• DTC_TEST_RUN_ID=issue-164-cache-boundary uv run --frozen python manage.py test core.tests.test_development_seo.DevelopmentRobotsAndSitemapTests core.tests.test_development_seo.DevelopmentResponsePolicyTests.test_credential_bearing_early_response_fails_closed --noinput — 4 passed.
• DTC_TEST_RUN_ID=issue-164-browser-cookie DJANGO_ALLOW_ASYNC_UNSAFE=true uv run --frozen pytest playwright_tests/test_historical_registration_totals.py::test_public_zero_one_plural_and_omitted_states_are_private_and_responsive -q — 2 passed (desktop and mobile).
• make check-seo — 76 passed; make check-links — 7 passed.
• make django-check, make migrations-check, make lint, make format-check, and make typecheck — all passed.
• Changed-file ruff format/check, mypy, and git diff --check — all passed.

The focused Django regression matrix covers production and development GET/HEAD responses with
authorization, session, CSRF cookie/header, unknown cookie, preview-token, and management-token
credentials, plus both documented anonymous preference cookies.

Fresh v4 verification artifacts

The plan was recomputed from the exact pinned base/head after the v3 browser regression was fixed:

• Plan: .tmp/verification/issue-164-v4/verification-plan.json, file SHA-256
  68c765af35706193f23073cc05d04f090ea28f88f307cb8327cb3dd1b26634a5;
• canonical plan digest in the report: 32e8bf9e5ce6fe9e30fc49e252d27f60d7de98480b1f343d30b3be68f303554d;
• graph SHA-256: 4127118b4467a75b40b8dbadc4050b5b2dc3762b292aca93cd9309a4593bca19;
• policy version: 4; profile: full; direct node: shared.core; downstream nodes: none;
• verification state SHA-256: eb23ae6b4621adc85d701b9da014ec3cac318fc4117fca0bd95459f64da680f9;
• report: .tmp/verification/issue-164-v4/verification-report.json, file SHA-256
  3f6a4dd1391e31f1f2c3bc51dcc5072f367c1f8dfa55bb4f9bc66aec1d437214;
• report verdict: pending_independent_tester.

Fresh evidence IDs and counts:

• quality: c0cbd17c57b8a219d6d9c3a6fe962df77f39a147cbdadb873924464a0f196f76 — 531/531;
• Django: 07a9a8e2deec6ba060c4936c424c91296f2119230743529fc4650f147b121130 — 2,100 passed,
  8 expected skips, 2,108 assertions;
• Playwright: 3e699ecbb906655cfe45cf91cbf3112771e398ec8c0d2801efe3c5ba68fb2515 — 181/181;
• container: 31a04b015987eb12022b2696f2e0e0678355ef2be40dc51861f566e6e5eac — 12/12 assertions;
• evidence/control validation: 931a5ab0fc24b8c3f31a4ea9876ce71aeaba6896c02d5a4f7bee8217d680fd57 — 531/531;
• selector: e9a3495aa86f01ac0006cc3d0dafccf71473fbd3e71bb2141911a1cf23739cab — 1 plan assertion;
• compatibility and content-invariant buckets are explicitly not applicable outside the impact
  closure.

The v4 plan has render impact from core/views.py and requires eight Chromium desktop/mobile
captures. Screenshots are intentionally skipped in the engineer phase (independent_tester_required);
no engineer screenshot is presented as independent evidence. The tester must recompute the plan
with consumer=tester, rerun as needed, capture and inspect all eight required views under the
project-local .tmp/screenshots/, and publish the tester-final report. No live production smoke was
run; that remains outside this issue's authority and the #160 handoff.

The candidate is now frozen uncommitted for that independent tester handoff.

[alexeygrigorev]

QA Review — PASS (independent tester, v5)

The frozen #164 candidate passes the independent tester gate. The candidate remains
uncommitted and no deployment or live-production probe was performed.

Candidate and verification identity

• Base/head: 754bc24cb0510c9894a07686fae925935dcaf754 / 754bc24cb0510c9894a07686fae925935dcaf754 (worktree overlay).
• Changed-file SHA-256: core/middleware.py a995a0ee09796a14119be90b9c406503f5079d7ad67c0df24ee2d6683cf35691; core/tests/test_development_seo.py eeb688609268924c8340e1386934d3ba5109c89fe33bb770c1ee094109b033a8; core/views.py 3a9f5b165179b0581d960fd1dc1c7e404eed063214b8d3d1d3c7980734cd74c7.
• Tester plan .tmp/verification/issue-164-tester-v5/verification-plan.json: raw SHA-256 588305cdb9f7dd1308a50f8404c7ccc7fe0eb573f4b55af44c43b5176644effb; canonical plan digest 8279e5ec67b9238e699c3febed33ebb1b743d8582a9756e093f5f510f0f6d3db.
• Graph 4127118b4467a75b40b8dbadc4050b5b2dc3762b292aca93cd9309a4593bca19; policy 4; source manifest ff84aae340f39ec40334272ee1e8558bb5f129594677a805cba5a49cded740b7; verification state eb23ae6b4621adc85d701b9da014ec3cac318fc4117fca0bd95459f64da680f9.
• Final tester report .tmp/verification/issue-164-tester-v5/verification-report.json: SHA-256 e08a28747ac0b74c92da9026f26a258d379855382bf47b2f482d1984c717c434; verdict success.

Independent request matrix

Synthetic local request matrix artifact .tmp/issue-164-request-matrix.json SHA-256
8a2631aa7fbe2f2f8366cfd8c94cebb86d462aa5ac2a71b44f993ea2486f9495 passed (one Django
test, system checks clean). It covered both NOINDEX=False and NOINDEX=True, GET/HEAD,
all seven unsafe methods, six credential classes on both methods and settings branches,
the two anonymous preference cookies, and the unchanged ten-location sitemap index.

• Production robots GET body: 221 bytes, SHA-256 1859ed7f2c668b9f4c3fa70aeaa44583d2f57f4b46be54bab9cbff0ee40a6d40; HEAD body empty (e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855).
• Development robots body: 26 bytes, SHA-256 331ea9090db0c9f6f597bd9840fd5b171830f6e0b3ba1cb24dfa91f0c95aedc1.
• Sitemap GET body: 829 bytes, SHA-256 d8ea3c14cf4d959598371451546ffff833395a16c015789e93a1489dafca9749; HEAD body empty; ten canonical locations exact.
• Unsafe methods: 14 settings-branch cases returned 405, Allow: GET, HEAD, and no-store, max-age=0.
• Credential cases: authorization, session, CSRF cookie, unknown cookie, preview token, and management token all returned private, no-store, max-age=0 for GET/HEAD in both branches; documented preference cookies remained anonymous-safe.

Fresh automated evidence

• Selector: 1f287313844ec2d688d324eb8cf122a125d1f3a26b0673d05441ecde087c4b94 — 1 plan assertion.
• Quality contract: 8b1e8ba59b388ba996d95f9f256c9afffa75a2fc4e21b492014aaca35a1eee17 — 531/531.
• Evidence validation: d43474cb9efce35faebf586b0804e0b00de13257daa35cf1fe456236c17ceed9 — 531/531.
• Django: 74c6fdb9e41d48201e6b8f0fc1ad2ae00f7ce95e58f95e771bb8130b06e9693b — 2,100 passed, 8 expected skips, 2,108 assertions.
• Playwright full: adbce6af7412a4b93c92909699fc3d6d983138bdaafeb9ea3a4a2c1ddae34fad — 181/181 (make test-playwright).
• Container: 5c5a74c042b4cd173f951b6b84c704ac2188d4f1787b4bc072b0e2c94129abdf — 12/12 assertions.
• Compatibility and content-invariant buckets are explicitly not_applicable outside the impact closure.

Independent screenshots

The required Chromium matrix was captured and visually inspected under .tmp/screenshots/issue-164/:
home, not-found, courses, and Studio sign-in at 1440x900 and 390x844. Screenshot evidence
0b21d75ec415abe92d1d45451904701cb69d6d55b0ccc20bf87804f77626a241 records 8/8 independent
passes; machine output screenshots.json SHA-256 is
472603256270135ba19d66202643f30c29444815f5c7428a2ab41e5630f3d596, and all captures bind
to render digest aa537de44b68f1960c69fb073032abef02c3e3da30c3e4eef6445249eef804a5.

No source files were changed by the tester, and git diff --check is clean. Live production
smoke remains intentionally deferred until the separately authorized #160 readiness/deployment
gate.

[alexeygrigorev]

Product-manager acceptance — #164 (superseding candidate)

Accepted based on the superseding engineer handoff #164 (comment) and independent tester PASS #164 (comment). The prior candidate and v2/v3 evidence remain invalidated as documented by engineering.

The exact three-file candidate is authorized for commit:

• base 754bc24cb0510c9894a07686fae925935dcaf754;
• production robots.txt returns the exact approved body, text/plain, 200 for GET/HEAD, and the approved revalidation cache policy with no robots header;
• development still returns the exact Disallow: / body with development noindex behavior;
• unsafe methods return 405 with Allow: GET, HEAD and no-store behavior;
• authorization, session, CSRF, preview, management, and unknown credentials are private/no-store, while only the two documented preference cookies remain anonymous-safe;
• the sitemap contract is unchanged;
• request matrix, 181 full Playwright, 2,108 Django assertions, 531 quality/evidence assertions, container checks, and 8/8 independent screenshots all pass.

The engineer may now create one focused commit with body exactly Closes #164; do not alter the frozen bytes before committing.

[alexeygrigorev]

Orchestrator merge verification

The PM-accepted superseding candidate was committed exactly as 8cfa2ee048145635ded8d55dc157d51fb45dc02c (Closes #164) and merged/pushed without edits as 822bd243cbdf67a646cf997b4649b1ec636df606.

A post-merge focused run of core.tests.test_development_seo passed 24/24 on the exact pushed tree. The authoritative push workflow is 31857913389; on-call is observing it. No AWS or protected data access was used.

[alexeygrigorev]

Reopening: the accepted code is merged, but on-call deployment failed safely before mutation; live verification is still pending.