Return a valid development courses sitemap

[alexeygrigorev]

Evidence

A read-only audit of https://web.dtcdev.click at current main checked the sitemap index and each child path on the development host.

• /sitemap.xml returns 200 and lists the ten canonical production locations required by the existing sitemap contract.
• Nine development child paths return 200 XML: main, blog, podcast, books, people, events, wiki, docs, and faq.
• /sitemaps/courses.xml returns 500 text/html with X-Robots-Tag: noindex, nofollow instead of a valid XML sitemap.

The original audit incorrectly treated the production-origin <loc> values in the index as development request hosts; those canonical production URLs are intentional and remain unchanged. This issue is corrected to the reproducible development-host defect.

Scope

Make /sitemaps/courses.xml return a valid XML <urlset> on development for the current deployed database state, including the empty/no-course state, while preserving canonical production <loc> values, no-slash route policy, and production behavior. Reuse the existing _section_records("courses") and sitemap contract where possible; diagnose and fix the actual failure rather than adding fabricated course rows or weakening error handling.

Own only the smallest sitemap generation/routing code and focused tests needed for the courses section. Preserve the other nine sections and the index contract.

Non-goals

• No content import or course data population.
• No change to robots/indexing policy, canonical origin, DNS/AWS/CDN state, or production sitemap submission.
• No migration/model behavior or unrelated route/SEO redesign.

Acceptance criteria

• Development /sitemaps/courses.xml returns 200 application/xml for an empty database and for a database containing visible/hidden courses.
• The response is well-formed XML, contains only canonical production no-slash URLs, and does not expose private course/registration data.
• Development index and all other child sitemap routes remain unchanged and valid; X-Robots-Tag: noindex, nofollow remains present in development.
• Focused Django/contract tests reproduce the empty and populated cases and reject a 500/HTML response.
• uv/Make quality and applicable browser/SEO gates pass. This is an XML response-only fix; screenshots are not applicable.

[alexeygrigorev]

PM correction — READY

The scope is corrected to the reproducible defect: nine development child sitemap paths are healthy, while /sitemaps/courses.xml alone returns a 500 HTML response. The production-origin locations in the index are intentional canonical values and must not be changed. This issue is independent of the pending content/course owner decisions because it fixes the existing sitemap response contract without importing or mutating course data.

Engineer ownership is limited to sitemap generation/routing and focused tests. Diagnose the actual 500, preserve all canonical/noindex behavior, and do not add fallback/fabricated records. Screenshots are not_applicable for this XML-only response contract; run the repository's applicable uv/Make gates and hand off an uncommitted exact candidate for independent testing.

[alexeygrigorev]

Dependency correction

The development-host /sitemaps/courses.xml 500 is caused by the same database outage that makes /courses return 500 and /health/ready return 503 (database unavailable). It is not valid to fix this sitemap by adding a fallback or fabricated data. Recovery is tracked in #160; #159 remains unimplemented until readiness is restored, after which the route will be rechecked before any code change.

[alexeygrigorev]

PM disposition — BLOCKED BY #160

No engineer changes are authorized for this issue while the database is unavailable. The existing local sitemap implementation and focused contract pass with a migrated SQLite database; the deployed 500 is an infrastructure/readiness incident. Resume only after #160 produces exact redacted recovery evidence and the route is rechecked.

[alexeygrigorev]

Dependency recheck — still blocked by #160

The latest exact push run 31857913389 passed all application gates and publish, but deploy failed safely before mutation with web captured terminal counts differ; the dev host remains on 57a5367 and /health/ready is 503 because database/migrations are unavailable. See #160 on-call evidence.

No courses sitemap fallback or fabricated data is being added. Reopen implementation only after #160 restores readiness and the exact /courses and /sitemaps/courses.xml checks can be rerun.

[alexeygrigorev]

QA Review — PASS (closure verification at current main + live development host)

No code change was authorized for #159 (root cause was the #160 development RDS outage, recovered
2026-08-16), so this closure verifies the acceptance criteria at current main and on the live
development host. Screenshots: not_applicable — this issue is an XML-response-only contract
(its own criteria say so); no rendered page is in scope.

Release identity (all live evidence pinned to this)

https://web.dtcdev.click/health/live at 2026-08-18 06:02–06:16 UTC, pinned at start and end of
probing (no deploy transition during the window):

• version 20260817-064746-a9edeea
• source_sha a9edeea6bfb9299a6b20be9d069a56613ef3c198
• image_digest sha256:57de038d6c5aef96dbe50441d0e4d4e6b0d5df82b7706d0cc648cd3324ef75ee
• /health/ready is 200 with database: ok, migrations: ok (recovery confirmed)

Local tree: clean worktree at main 960334284893b4e338fcdd6d3395b11f2f098f1a (git status
clean, git diff --check clean). Because there is no uncommitted engineer handoff,
make verification-plan VERIFY_CONSUMER=tester has no base/head range to recompute: with
base=head=HEAD the classifier records changed_path_count: 0, reason: "diff_empty"
(.tmp/verification/qa-159-164-closure/ci-selection.json) and the planner rejects the empty range.
Provenance is therefore pinned by release identity and git SHAs above instead of a plan envelope.

Live development probes (release a9edeea)

• GET /sitemaps/courses.xml: 200, content-type: application/xml; charset=utf-8,
  X-Robots-Tag: noindex, nofollow, 1307 bytes, body SHA-256
  43815ed5417223e0f7833af0b9449c3b0dd81ce50f90ed12b914b21f25c26a1b. Well-formed
  <urlset> with 17 URLs: /courses hub, 15 visible course slugs, and the
  /courses/ai-dev-tools-zoomcamp/cohorts/ai-dev-tools-2026 cohort path — all
  https://datatalks.club/... canonical production no-slash URLs, no duplicates, no query or
  fragment, no private course/registration data (marker scan clean).
• GET /sitemap.xml: 200 application/xml, 829 bytes, SHA-256
  d8ea3c14cf4d959598371451546ffff833395a16c015789e93a1489dafca9749; validated at main with
  content.sitemap_contract.validate_sitemap_index — equals EXPECTED_SITEMAP_LOCATIONS exactly
  (10 sections).
• All other nine children return 200 application/xml with X-Robots-Tag: noindex, nofollow
  (sizes/SHA-256 recorded under .tmp/qa-159-164/): main 358
  63ccb4697b008fe21105887f2114447d86a9d42391086b3d8f19838b35945794, blog 6743
  2bb66f27ae5fdf47b2acf9193b41096cb1270764333d314f5ffa9e64172c62cb, podcast 26798
  e64b31bfb3c30c4b1cdd12b742f897eaf6105dd9bbba1fe75db18a654357abe8, books 12344
  c32891efbd84f3e83148489b898c1a1d9825a4aa0bca7a13a4f668144e9645bc, people 30591
  a7af25d9f72f33478892e7dbc67b7c7fbaf92fa4cc2ae458411360ca2f0611d4, events 53270
  3218c16a0746001add7ea1457ba90baa1973c40690f527ab4d8e6bc99343bb02, wiki 20764
  c12af37772945e0ec5cbb4946aa98d1154643fbb3c5d52d07649f1e7ef42826b, docs 253
  8ffaa317d89e70c8e5d12f02ea08349e4daede5f3149330248c788449f3cb80e, faq 609
  2a27ee01e9d64fd3a6588894a3b2993b9cf753230590688bdcc738281dd16acc (bytes; SHA-256 in that
  order after each size). HEAD /sitemap.xml, /sitemaps/courses.xml, and /sitemaps/wiki.xml
  are 200 with matching content type/length and empty transfer bodies.
• Spot checks: /courses 200 with canonical https://datatalks.club/courses;
  /courses/de-zoomcamp-2026 and the cohort path render 200 with canonical production links;
  an unknown course slug is 404.

Local verification at main (9603342)

• DJANGO_SETTINGS_MODULE=website.settings.test DTC_TEST_RUN_ID=qa-159-sitemap-contract uv run --frozen python manage.py test content.tests.test_sitemap_contract content.tests.test_public_routes_and_seo.PublicRouteAndSeoTests.test_every_section_sitemap_entry_is_a_unique_canonical_public_200 --noinput
  — 4 tests, OK. test_every_section_sitemap_entry_is_a_unique_canonical_public_200 exercises
  /sitemaps/courses.xml through the live index against a migrated SQLite test database with an
  empty Course table (empty-database case) and rejects any 500/HTML response by asserting 200,
  parsing the XML, and requiring canonical production locations that render 200.
• DTC_TEST_RUN_ID=qa-159-164-robots uv run --frozen python manage.py test core.tests.test_development_seo.DevelopmentRobotsAndSitemapTests --noinput
  — 3 tests, OK (index exposure on development).
• Independent empty + populated matrix on a disposable migrated test database:
  uv run --frozen python .tmp/qa-159-164/scratch_courses_sitemap.py (script SHA-256
  6a4ae5fe5e5cfa4fd99651d92cb775ac59fc30fb4d7310d18afdebf41ef07491, log
  .tmp/qa-159-164/scratch-final.log SHA-256
  bf571f1058787e60c1b25e233a5c1f578a5f7d291ad6e826c196a609f3f52048) — 41/41 checks passed:
  empty database returns exactly the hub + cohort entries as 200 application/xml; with visible
  and hidden Course rows the visible slugs appear in slug order as canonical no-slash production
  URLs, the hidden slug is excluded, no duplicates, no private markers, index and a sibling
  section remain valid.
• make check-seo — 76 passed. make check-links — 7 passed.
• make django-check — clean; make migrations-check — no changes; make lint,
  make format-check (543 files), make security-check — pass.

Checkbox dispositions

Ticked 1–3 (verified live and locally as above). Left unticked, precisely:

• Criterion 4 (focused tests reproduce empty and populated cases): the repository tests cover
  the empty case and reject 500/HTML, but no committed test covers the populated visible/hidden
  courses case; my 41-check scratch harness reproduces it, and a durable test would require an
  authorized code change (outside this closure's no-code-change disposition).
• Criterion 5 (uv/Make quality and applicable browser/SEO gates): every sitemap/SEO/browser/lint/
  format/security/migration gate above is green, but make typecheck at main fails with 7
  union-attr errors in content/tests/test_article_faq.py — a file introduced by later commit
  e96e256 ("Recover the article FAQs the migration dropped"), after the Return a valid development courses sitemap #159 diagnosis, and
  unrelated to sitemaps (content/public_views.py is mypy-clean). This regression needs its own
  fix and equally blocks any current closure's repository gates.

Verdict: PASS. The #159 defect (development /sitemaps/courses.xml 500) is resolved on the
live development host at release a9edeea and the contract holds at current main; the two unticked
criteria are documented above (one durable-test gap, one unrelated repo-wide typecheck
regression), not sitemap failures.

[alexeygrigorev]

Product Acceptance — ACCEPT

Independent PM verification at current main (9603342, clean worktree) and against the live development host pinned to release a9edeea (source_sha a9edeea6bfb9299a6b20be9d069a56613ef3c198), performed 2026-08-18 after the tester PASS in #159 (comment).

Visitor perspective (live, my own probes)

• GET /sitemaps/courses.xml is 200 application/xml with X-Robots-Tag: noindex, nofollow: a well-formed <urlset> of exactly 17 canonical production no-slash URLs — the /courses hub, 15 visible course slugs, and the ai-dev-tools-2026 cohort path. No duplicates, query strings, fragments, development host, or private/registration data.
• GET /sitemap.xml is 200 application/xml listing exactly the ten canonical production section locations; the nine sibling sections all serve 200 application/xml under development noindex.
• /courses renders 200 with the canonical production origin; an unknown course slug is a safe 404.
• /health/ready is 200 with database: ok and migrations: ok, so the populated state is genuinely database-backed, not a fallback.

Disposition of the unticked criteria

• Criterion 4 (committed empty + populated tests): genuinely unmet as written. The committed suite covers the empty case and rejects 500/HTML; no committed test covers populated visible/hidden courses (the tester's 41-check harness was scratch-only). I judge closure acceptable with a bounded follow-up rather than keeping this P0 open: the reported defect is verifiably resolved at two release identities, the root cause was the Restore development database readiness for web.dtcdev.click #160 infrastructure outage requiring no code change, and returning this issue to engineering would mean authorizing new code that is exactly the bounded test work now tracked in Commit the populated courses sitemap contract test (visible and hidden courses) #189 through the normal lifecycle. Checkbox stays unticked; the gap is owned, not skipped.
• Criterion 5 (quality gates): every issue-specific gate is green (check-seo 76, check-links 7, django-check, migrations-check, lint, format-check, security-check, and the focused sitemap/SEO selections). make typecheck is red at main with exactly 7 union-attr errors in content/tests/test_article_faq.py — I reproduced this locally. That regression was introduced by the later, unrelated commit e96e256 ("Recover the article FAQs the migration dropped") after this issue's diagnosis, touches no sitemap path, and is owned by the on-call remediation for e96e256. Per the process the red gate is attributed to the introducing change and must be fixed there; it is not a Return a valid development courses sitemap #159 failure and does not block closure. Owner attention: no open issue currently tracks that regression (I searched); on-call should attribute and file it.

Screenshots not_applicable is valid: this issue's own criteria define an XML-response-only contract.

The root cause (development RDS credential rotation outage tracked by #160, recovered 2026-08-16) is documented end to end; no fabricated data or weakened error handling was introduced.

[alexeygrigorev]

Closing #159.

• Defect: development /sitemaps/courses.xml returned 500 text/html. Root cause was the Restore development database readiness for web.dtcdev.click #160 development database outage (RDS master-credential rotation desync), not an application defect; recovery on 2026-08-16 restored the route with no code change, and the contract has since been live-verified at releases 57a5367 (post-recovery) and a9edeea.
• Tester PASS: Return a valid development courses sitemap #159 (comment) (criteria 1–3 ticked with pinned release identity, digests, and local runs at main).
• PM acceptance: Return a valid development courses sitemap #159 (comment) (independent live probes of the 17-URL sitemap, index, siblings, /courses, safe 404, and readiness).
• Follow-up: Commit the populated courses sitemap contract test (visible and hidden courses) #189 commits the populated visible/hidden courses sitemap test (criterion 4 recorded as unmet-but-owned there).
• Non-blocking external item: the make typecheck regression from e96e256 in content/tests/test_article_faq.py is owned by on-call remediation and is not attributable to this issue.