Migrated website from python to nodejs and merged HAR docs

[max-ostapenko]

• all the httparchive.org content managed by Astro
• most of the website is served as static, moved dates and metrics to API endpoints
• merged har.fyi content under docs/

There were many duplicated and custom operations, like JS management and resources versioning, that are taken care of by framework anyway.

[github-advanced-security]

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

• The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
• Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
• You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

[max-ostapenko]

@tunetheweb closed all TODO's.

[max-ostapenko]

@tunetheweb here are the summarized changes.

Preview deployment: https://httparchive--staging-supndlqf.web.app/
Hosting console: https://console.firebase.google.com/project/httparchive/hosting/sites/httparchive/channels/staging

1. Hosting

• Migrated from App Engine to Firebase Hosting: Converted legacy Python/Flask App Engine setup to a modern, serverless static architecture powered by Astro + Firebase Hosting.
• Removed App Engine Code: Deleted app.yaml, server.js, Python server/ files, Jinja templates/, and express dependency.
• Added Firebase Hosting Configs:
  • .firebaserc: Configured project mapping to httparchive.
  • firebase.json: Configured static serving out of dist/, cleanUrls: true, trailingSlash: false, font caching (/static/fonts/**), security headers (CSP, HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy), and default emulator port 8080.

2. Custom 404 & SEO Parity

• Custom 404 Page: Added src/pages/404.astro matching original App Engine 404 template parity (issue link, nav buttons, magnifying glass illustration).
• Starlight Config: Set disable404Route: true in astro.config.mjs so Astro compiles the custom 404 page directly to dist/404.html.
• Enhanced SEO Timestamps: Updated src/layouts/Base.astro to apply last_updated.json date_published and date_modified metadata across all /docs/* routes.

3. Development Workflow

• npm start: Builds site and starts native Firebase Hosting emulator (npx firebase-tools emulators:start --only hosting).
• npm run stage: Deploys to Firebase Hosting staging preview channel (npx firebase-tools hosting:channel:deploy staging).
• npm run deploy: Deploys to Firebase Hosting live production channel (npx firebase-tools deploy --only hosting).
• Removed Redundant Scripts: Removed tools/scripts/deploy.sh and obsolete preview / testdeploy npm commands.

4. GitHub Actions

• firebase-preview.yml: Added workflow to automatically build and deploy preview channels for all incoming Pull Requests using FirebaseExtended/action-hosting-deploy@v0.
• deploy.yml: Updated workflow to deploy to Firebase Hosting live channel on push to main.
• Automated Test Scripts: Updated tools/scripts/run_and_test_website.sh and run_and_test_website.bat to run tests against the Firebase Hosting emulator (100% test pass rate: 54/54 passed).
• Lighthouse CI: Updated tools/scripts/set_lighthouse_urls.sh diff paths to scan src/pages/ and public/sitemap.xml.

5. Code Generators (tools/generate/)

• Extracted Shared Helpers: Moved map_to_astro_page() to tools/generate/shared.js.
• Removed Dead Code & Dependencies: Uninstalled recursive-readdir and removed dead find_asset_files() function.
• Modernized Node APIs: Replaced recursive-readdir with native fs.promises.readdir({ recursive: true }) in generate_timestamps.js and replaced deprecated .substr() in generate_sitemap.js.
• Cleaned last_updated.json: Pruned legacy /static/ asset entries from config/last_updated.json (reduced lines from 403 to 203), focusing purely on content pages, reports, and documentation.

6. UI

• Subdomain Lens Detection: Added client-side subdomain detection in src/pages/reports/[report_id].astro and src/js/report.js so subdomains (e.g. wordpress.httparchive.org) automatically pre-select their corresponding lens.

7. Route Mapping (Flask vs. Astro)

Route Path	Production main (Flask)	Migrated Version (Astro / Firebase)	Output Artifact
/	server/routes.py: index() -> templates/index.html	src/pages/index.astro	dist/index.html
/about	server/routes.py: about() -> templates/about.html	src/pages/about.astro	dist/about/index.html
/faq	server/routes.py: faq() -> templates/faq.html	src/pages/faq.astro	dist/faq/index.html
/reports	server/routes.py: reports() -> templates/reports.html	src/pages/reports/index.astro	dist/reports/index.html
/reports/[report_id]	server/routes.py: report() -> templates/report/report.html	src/pages/reports/[report_id].astro	dist/reports/[report_id]/index.html
/reports/techreport/landing	server/routes.py: techreportlanding() -> templates/techreport/landing.html	src/pages/reports/techreport/landing.astro	dist/reports/techreport/landing/index.html
/reports/techreport/tech	server/routes.py: techreport() -> templates/techreport/techreport.html	src/pages/reports/techreport/tech.astro	dist/reports/techreport/tech/index.html
/reports/techreport/[page_id]	server/routes.py: techreportlanding()	src/pages/reports/techreport/[page_id].astro	dist/reports/techreport/[page_id]/index.html
/docs/*	N/A	src/content/docs/docs/* (Starlight MDX)	dist/docs/*
/404	server/errors.py -> templates/error/404.html	src/pages/404.astro	dist/404.html
/sitemap.xml	server/routes.py -> templates/sitemap.xml	tools/generate/generate_sitemap.js	public/sitemap.xml
/robots.txt	server/routes.py: static_from_root()	public/robots.txt	dist/robots.txt
/favicon.ico	server/routes.py: favicon()	public/favicon.ico	dist/favicon.ico

---

8. HTTP Headers

Production Talisman headers (server/csp.py) were ported 1:1 into firebase.json for all routes (/**).

9. HAR

Migrated into content/docs/docs.

10. Server logic moved to API

See HTTPArchive/tech-report-apis#128

Note: I think it may be more ergonomic to merge API here, keeping closer to frontend.

[tunetheweb]

@max-ostapenko finally coming round to look at this (apologies for the delay!) and stumbling at the first hurdle:

$ npm i
npm error code ERESOLVE
npm error ERESOLVE could not resolve
npm error
npm error While resolving: starlight-llms-txt@0.10.0
npm error Found: astro@7.0.3
npm error node_modules/astro
npm error   dev astro@"^7.0.3" from the root project
npm error   peer astro@"^7.0.0-alpha.0" from @astrojs/mdx@7.0.0
npm error   node_modules/@astrojs/mdx
npm error     @astrojs/mdx@"^7.0.0" from @astrojs/starlight@0.41.1
npm error     node_modules/@astrojs/starlight
npm error       dev @astrojs/starlight@"^0.41.1" from the root project
npm error       2 more (starlight-links-validator, starlight-llms-txt)
npm error   3 more (@astrojs/starlight, astro-expressive-code, starlight-links-validator)
npm error
npm error Could not resolve dependency:
npm error peer astro@"^6.0.0" from starlight-llms-txt@0.10.0
npm error node_modules/starlight-llms-txt
npm error   dev starlight-llms-txt@"^0.10.0" from the root project
npm error
npm error Conflicting peer dependency: astro@6.4.8
npm error node_modules/astro
npm error   peer astro@"^6.0.0" from starlight-llms-txt@0.10.0
npm error   node_modules/starlight-llms-txt
npm error     dev starlight-llms-txt@"^0.10.0" from the root project
npm error
npm error Fix the upstream dependency conflict, or retry this command with --force or --legacy-peer-deps to accept an incorrect (and potentially broken) dependency resolution.
npm error
npm error

I also see some merge conflicts.

Can you look at resolving this and will have a look.

[max-ostapenko]

@tunetheweb resolved.

[tunetheweb]

Not gone through it all, but going to stop here given the issues with the /docs site. As started in the other comment, I think we should split this PR.

I also see some styling differences:

[tunetheweb]

Out of interest why are these not needed? They are used to provide a lastmod to /sitemap.xml

[tunetheweb]

Suggested change

The test agents are run from [Google Cloud regions](https://cloud.google.com/compute/docs/regions-zones) across the US. Each URL is loaded once with an empty cache ("first view") for normal metrics collection and again, in a clean browser profile, using [Lighthouse](https://developers.google.com/web/tools/lighthouse). The data is collected via a [HAR file](https://en.wikipedia.org/wiki/.har). The HTTP Archive collects these HAR files, parses them, and populates [a public dataset in BigQuery](docs/guides/getting-started/).

The test agents are run from [Google Cloud regions](https://cloud.google.com/compute/docs/regions-zones) across the US. Each URL is loaded once with an empty cache ("first view") for normal metrics collection and again, in a clean browser profile, using [Lighthouse](https://developers.google.com/web/tools/lighthouse). The data is collected via a [HAR file](https://en.wikipedia.org/wiki/.har). The HTTP Archive collects these HAR files, parses them, and populates [a public dataset in BigQuery](/docs/guides/getting-started/).

[tunetheweb]

Suggested change

	The HTTP Archive dataset is available publicly on BigQuery. Check out [Getting Started Accessing the HTTP Archive with BigQuery](docs/guides/getting-started/).
	The HTTP Archive dataset is available publicly on BigQuery. Check out [Getting Started Accessing the HTTP Archive with BigQuery](/docs/guides/getting-started/).

[tunetheweb]

Why map.png instead of web.png as it was before? That's not a great name either to be honest so maybe hero.png is better?

[tunetheweb]

This doesn't auto refresh when editing code. Should we use astro:dev which includes a server so doesn't need firebase?

Or is there some combo of both if you want to test e2e to include firebase?

[tunetheweb]

Do we really need the double nesting of docs in content/docs/docs/? I know it isn't surface to the user in the URL but it's still quite confusing...

[tunetheweb]

I see a double hamburger menu in /docs:

Additionally, the original hamburger menu differs from the rest of the site (Reports is not, and cannot be expanded):

I think we're biting off too much in one go here and should split this PR into 3 PRs:

• Migrate from python to node
• Migrate from App Engine to Firebase
• Integrate har.fyi