feat: add --stop-on-rate-limit and --retry-on-rate-limit flags for dir mode

[juliosuas]

Summary

Adds two new flags to the dir mode to handle HTTP 429 (Too Many Requests) responses gracefully. Addresses #363.

Flags

--stop-on-rate-limit

When a 429 response is received, the scan stops gracefully with a clear error message.

gobuster dir -u https://target.com -w wordlist.txt --stop-on-rate-limit

--retry-on-rate-limit

When a 429 response is received, gobuster waits for the duration specified in the Retry-After header (or 5 seconds by default) and retries the same request.

gobuster dir -u https://target.com -w wordlist.txt --retry-on-rate-limit

The two flags are mutually exclusive — setting both will produce an error.

Use Case

In bug bounty and penetration testing scenarios, targets commonly enforce rate limiting. Currently, gobuster keeps hammering the target with requests even after receiving 429 responses, which:

• Wastes time on requests that will be rejected
• May get the tester IP-banned
• Generates noisy logs on the target

With --retry-on-rate-limit, testers can slow down automatically and respect the target rate limits while still completing the scan. With --stop-on-rate-limit, testers can abort early and adjust their approach (e.g., reduce threads, add delay).

Implementation Details

• Minimal and focused: only 5 files changed, ~78 lines added
• Handles Retry-After header as both seconds (integer) and HTTP-date formats
• Uses context.CancelFunc on Progress to allow plugins to signal graceful stop
• Added config string output so the flags show in the startup banner
• No new dependencies

[juliosuas]

Rebased on upstream dev branch and resolved merge conflicts. The conflicts were in cli/dir/dir.go, gobusterdir/gobusterdir.go, and gobusterdir/options.go — upstream added regex filtering features in adjacent code blocks. Both feature sets (regex + rate-limit flags) are now cleanly preserved. Single commit, clean diff against dev.