Evaluating URL-only phishing detection, honestly.

Summary

• A shortcut, found and removed. In one dataset every legitimate URL was a bare homepage. A no-ML rule exploiting that scores F1 0.996 there. We excluded those signals from the main model.
• Better than the current demo on familiar data. On websites held out from the same dataset, the hostname model reaches ROC-AUC 0.919 against 0.848 for the model in today’s demo.
• Generalisation is the open problem. On a separate dataset the advantage shrinks to 0.741 against 0.717, with overlapping uncertainty, and a threshold tuned for 1% false positives produced 6.2%.

Data

Both datasets are published under CC BY 4.0, which permits this use with attribution. We use only each dataset’s URL text and label; supplied features derived from page content are ignored, because a URL-only detector cannot see them.

Cleaning removed URLs that the app’s own validator rejects, exact duplicates, and URLs with conflicting labels. Neither publisher documents where its URLs came from.

Method

1. Split by owner, not by row. URLs are grouped by registrable domain using the Public Suffix List, then assigned 70/15/15 to training, validation and test. No domain appears in two splits.
2. Choose on validation only. Model family, settings and decision thresholds were selected on the validation split. Test sets were scored once, afterwards.
3. Test across datasets. Models trained on one dataset were also tested on the other, after removing every domain seen in training or validation.
4. Report uncertainty honestly. Ranges are 95% intervals from a bootstrap that resamples whole domains, because a few domains contribute many URLs.

Three model families were compared: logistic regression on hand-designed lexical features (the baseline), gradient-boosted trees on the same features, and logistic regression on character 3–5-grams. The character n-gram model won on validation in both experiments.

Finding: a dataset shortcut

Before training anything, we profiled the URLs. PhiUSIIL’s legitimate class turned out to be built entirely from homepages:

A rule with no machine learning—legitimate if https, www and no path—reaches F1 0.996 on PhiUSIIL’s held-out domains and only 0.754 on the Hannousse dataset, where it flags 84% of legitimate URLs. A model that sees those properties would learn the dataset, not phishing. Our main model therefore sees only the hostname, with a leading www. removed.

Results

Positive class is phishing. “Recall at 1% FPR” uses the threshold chosen on validation to allow 1% false positives; the false-positive rate it actually produced on each test set is shown next to it.

Hostname model, trained on PhiUSIIL

Full-URL model, trained on Hannousse

The Hannousse dataset has realistic legitimate URLs (with paths, both http and https), so here the model may read the whole URL. Its test set is small, so the intervals are wide.

* The current demo model gives its maximum score to more than 1% of legitimate validation URLs, so no threshold reaches 1% false positives and it flags nothing at that operating point.

The current demo, as it actually behaves

At the app’s own default cut-off (a model score of 0.5), the demo model catches 10.6% of phishing URLs on held-out PhiUSIIL domains, and 38.2% on the Hannousse dataset while flagging 13.4% of legitimate URLs. This is why the demo is labelled a demonstration.

Limitations

• Distribution shift. PhiUSIIL’s legitimate URLs are popular homepages; real-world legitimate links are far more varied, so false-positive rates in practice are likely higher than on PhiUSIIL.
• A few domains dominate. One hosting domain contributes 1,550 PhiUSIIL test URLs; one site is 18% of the Hannousse test set. That is why intervals are wide.
• No time dimension. Neither dataset has timestamps, so drift over time is unmeasured, and phishing infrastructure changes quickly.
• Text only. A URL-only model cannot recognise a compromised legitimate site, content on reputable hosting platforms, or a convincing new domain.
• Unknown provenance. Neither dataset documents its URL sources, so label noise cannot be quantified.

What comes next

The next step is improving cross-dataset behaviour, not the in-dataset score: training on both datasets together, weighting evaluation per domain, setting thresholds for each deployment context, and adding a licensed phishing source with timestamps to measure drift. Until then, the candidates stay experimental and the public demo keeps its warning labels.

Try the URL dissector · Open the research demo (opens in a new tab) ↗