Many links

Every page below is written by generate-assets.sh and is not in the repository. Run the script first, or these links all answer with a 404 — which is its own kind of test.

Hubs

Each hub links to that many target pages, and every target links on to the next one, so a crawl that follows everything walks the whole set rather than stopping one level down.

• 100 links — well inside any budget
• 500 links — past the default budget of 400 pages
• 1,000 links — two and a half times the budget, on one page
• 5,000 links — a single document of several megabytes of anchors

Shapes that are worse than a long list

• A chain — every page links to exactly one more, a thousand deep
• A calendar — a next-month link that never runs out, which is how a crawl becomes infinite
• Faceted search — the same page under every combination of four parameters
• Duplicates — one page linked five hundred times under five hundred URLs that mean the same thing