Consider only accepting 80/443 connections from Cloudflare

[jpmckinney]

• We should first check for evidence of any IPs circumventing Cloudflare across our origins.

[jpmckinney]

@RobHooper wrote on Slack:

The only thing I think this would break is the placeholder site on the server hostnames (such as ocp21.open-contracting.org) -  these endpoints cannot be proxied through Cloudflare as it breaks SSH access. Since it is not a formal website, I think we are OK to lose this.

• We would need an alert / prompt to update the IP list when Cloudflare publishes a new endpoint.
• Some Pingdom checks may be circumventing Cloudflare to perform a more thorough test of the hosting stack such as ensuring IPv6 is working behind the scenes. This would just need thinking about.

[jpmckinney]

Claude findings (slightly edited):

Requests that reach an origin directly without a matching ServerName land in the default vhost log: credential harvesting and known-exploit probing:

origin	role	default-vhost hits	notable paths
ocp19	docs	670	/.env, /.env.production
ocp20	prometheus	1088	/wp-content/plugins/hellopress/wp_filemanager.php, /this_is_a_new_hello_world.php
ocp21	cms	554	/.env, /.env.local, /.env.bak, /.env.backup
ocp23	kingfisher-main	435	/.env, hellopress/wp_filemanager.php
ocp25	dream-bi	0	no default access log
ocp28	cove	426	/.env, /env
ocp29	registry	1600	/.env, /.env.local

Agents include feroxbuster/2.13.1, crusader-worker/1.0, CyberConvoyScout, and research-scan/1.0.

Those counts are only the requests that failed to match a vhost. A request sent directly with correct SNI is logged identically to a proxied one, because mod_remoteip rewrites the client address and the log format never records the peer.

Need to update log format to use %{c}a to expose the underlying connection peer. [done]

TLS certificates

Restricting 80/443 to Cloudflare ranges will break certificate renewal on any origin using mod_md with TLS-ALPN-01 — ocp21's vhost declares Protocols h2 http/1.1 acme-tls/1, and Let's Encrypt must reach the origin directly for that. Cloudflare terminates TLS and cannot proxy the validation.

• Options, roughly in order of preference:

1. Authenticated Origin Pulls — require a Cloudflare client certificate via SSLVerifyClient. Does not rot as Cloudflare's ranges change, but still needs an ACME exemption or a move to DNS-01.
2. Switch mod_md to DNS-01, then allowlist Cloudflare ranges freely. DNS is already managed in Cloudflare.
3. Cloudflare Origin CA certificates on the origins — no ACME validation at all, but the origin cert is then only valid to Cloudflare, which complicates direct testing.

[jpmckinney]

I updated Apache logs so that we can distinguish Cloudflare-proxied and unproxied requests (except registry, which is mid-scrape).

Now can run, e.g. on ocp21:

awk '{print $NF}' /var/log/apache2/corporate/access.log | sort -u | head

To replicate:

curl -sS -o /dev/null -D - --resolve www.open-contracting.org:443:139.162.211.65 \
  https://www.open-contracting.org/