# =================================================================
# Block backlink/SEO-intel spies + aggressive scrapers (no referral,
# pure parasites). Impolite ones also get 403'd at .htaccess.
# =================================================================
User-agent: serpstatbot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: ImagesiftBot
Disallow: /

# =================================================================
# AI answer-engine crawlers — EXPLICITLY WELCOME (2026-05-31 reposition).
# Citation in ChatGPT / Perplexity / Claude / Google AI Overviews / Apple
# Intelligence is the growth channel. Let them read everything.
# =================================================================
User-agent: GPTBot
Disallow: /cruise-calendar.php?*sort=
Disallow: /cruise-calendar.php?*nights=
Disallow: /cruise-calendar.php?*line[
Allow: /cruise-calendar.php?line=
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: CCBot
Allow: /

User-agent: meta-externalagent
Allow: /

# =================================================================
# Default policy — all other bots welcome with a 5s crawl delay.
# =================================================================
# CRUISE-CALENDAR FACETS (rewritten 2026-08-31). The blanket `Disallow: /cruise-calendar.php?`
# was doing the opposite of what it looked like. The page already defends itself: a SINGLE-facet
# view (?region=Alaska, ?port=seattle, ?tier=Luxury) is self-canonical and indexable, and any
# COMBINATION (&line[]=, &sort=, &nights=) already returns `noindex,follow` and canonicals back to
# the bare page. Disallow cannot see any of that -- a blocked URL is never fetched, so the noindex
# is never read, and Google is free to index the URL from links alone with no content at all. It
# also blocked the 192 curated landing pages our own sitemap lists, so robots.txt and sitemap.xml
# were telling Google opposite things. Now the combinatorial parameters are named explicitly and
# the curated facets are crawlable, which is where the real queries live ("alaska cruises").
#
# 2026-09-22, Rob: "open the pages up then". ONE CRUISE LINE joins region / port / tier as a
# crawlable facet: `?line=Carnival` is 22 named pages, the same shape as the 18 region pages, and
# it is where "carnival cruises" (1.83M/mo) and "viking river cruises" (90.5k) land. The BRACKETED
# `line[]=` form stays disallowed - that is the combinatorial space the July botnet walked, and the
# .htaccess bot gate was opened for the bare form only, on the same day. A Disallow and an
# indexable page must never disagree again.
User-agent: *
Disallow: /go.php
Disallow: /cruise-calendar.php?*sort=
Disallow: /cruise-calendar.php?*nights=
Disallow: /cruise-calendar.php?*line[
Allow: /cruise-calendar.php?region=
Allow: /cruise-calendar.php?port=
Allow: /cruise-calendar.php?tier=
Allow: /cruise-calendar.php?line=
Allow: /
Crawl-delay: 5

Sitemap: https://fantasize.net/sitemap.xml