# Robots.txt for Sprawl
# Welcome every search engine and AI crawler to all marketing content,
# but keep them out of the private/app routes.
#
# One `User-agent: *` group on purpose: robots.txt is most-specific-group-wins,
# so a named bot with its own group would NOT inherit rules from a separate `*`
# group. A single `*` group means the Allow + Disallow rules below apply to
# EVERY crawler — Googlebot, GPTBot, ClaudeBot, PerplexityBot, and the rest.
#
# WHY /chat, /sign-in AND /sign-up ARE NOT DISALLOWED (2026-08-15)
# ----------------------------------------------------------------
# They used to be, and it backfired. All three are deliberately PUBLIC
# (proxy.ts `isPublicRoute`: "Chat page - viewable without auth"), so they
# answer an anonymous crawler with a real 200, the root layout's
# `robots: { index: true, follow: true }`, and `<link rel="canonical"
# href="https://sprawl.to">`.
#
# Disallow does not mean "remove from the index" — it means "do not fetch".
# A URL already in the index therefore STAYS there, frozen, because the
# crawler is forbidden from fetching the page that would tell it otherwise.
# That is exactly what happened: searching for Sprawl surfaced /chat with the
# pre-2026-07-15 positioning ("AI Agent for Every Integration… 40+ apps"),
# months after the site was rewritten, and the Disallow was what kept it
# there. The canonical tag that would fold /chat into the homepage was
# unreachable for the same reason.
#
# So they are crawlable now, and the canonical does the consolidating. If a
# page here should genuinely stay out of the index, give it `noindex` in its
# own metadata — a directive a crawler has to FETCH the page to obey — and
# leave it crawlable. `robots-txt.test.ts` fails the gate if a public page
# route reappears below.

User-agent: *
Allow: /
Disallow: /api/
Disallow: /settings
Disallow: /billing
Disallow: /scheduled
Disallow: /history

Sitemap: https://sprawl.to/sitemap.xml