Fix isDisallowed invalid URL handling

[AnandChowdhary]

Summary

• propagate undefined from isAllowed() through isDisallowed() for invalid/out-of-scope URLs instead of coercing it to true
• add regression coverage for invalid URL handling across both methods
• separate port-mismatch cases from explicitly disallowed URL cases in tests

Test plan

• git diff --cached --check
• added-line security scan: no findings
• independent Claude Code review: passed; no security concerns or logic errors
• npm test (44 passing, 100% coverage)
• npm audit --omit=dev (0 vulnerabilities)

Refs #41

[samclarke]

Sorry for taking so long to get to this!

Thinking about this, I'm wondering if it might make more sense to deprecate isDisallowed() instead of fixing it as it's easy to miss that it can return undefined.

Someone could easily do:

if (!robots.isDisallowed(url)) {

which could result in them crawling URLs that are not in scope and possibly getting their bot banned. What do you think?

It isn't an issue with isAllowed() as undefined will coerce to false so if they miss that it can return undefined, the worst case is some URLs not being crawled which should be noticed and won't risk them getting banned.