# Freewyld - StayFreewyld cabin rentals # https://freewyld.com # Allow all search engine crawlers User-agent: * Allow: / Allow: /agent/ # The agent plane, named file by file. These Allow lines are redundant against # "Allow: /agent/" above and that is the point: robots.txt is one of only two # places crawlers demonstrably read, so it is where the manifests have to be # listed. 26 days of hit logs say entry-point discovery does not work. Every # named AI crawler that has touched these surfaces arrived at a LEAF, via the # sitemap or the rel=alternate link, while /agent/index.json, reachable only # from itself, has never been read by one of them. Listing them here, and as # in the page head, puts them on # the two routes that actually work. # /llms.txt cabins, rates, policies, and the booking URL contract # /agent/index.json entry point # /agent/pages/index.json every agent page, with its markdown mirror Allow: /llms.txt Allow: /agent/index.json Allow: /agent/pages/index.json Disallow: /api/ Disallow: /_astro/ Disallow: /thank-you/ # Sitemap location Sitemap: https://freewyld.com/sitemap-index.xml # No blanket "Disallow: /*.json$" here on purpose. It used to sit alongside # "Allow: /agent/*.json$", so /agent/index.json and /agent/pages/index.json each # matched an Allow and a Disallow at once and every crawler resolved the tie its # own way. The only .json routes this site serves ARE those two manifests, so the # rule protected nothing: /api/ and /_astro/ are already blocked above by prefix. # If a data endpoint ever needs blocking, block that path, never all of *.json. # # AI/LLM crawlers are welcome. They inherit the * group above (e.g. ClaudeBot, # OAI-SearchBot, GPTBot, PerplexityBot); do NOT give them their own User-agent # groups (a dedicated group with only "Allow: /" would make them ignore the # * group's Disallow lines). The /agent/ Allow line above makes the intent # explicit: the /agent/ markdown mirrors and JSON manifests are for them.