# latchel.com: everything here is meant to be read, by people and by machines. User-agent: * Allow: / # Search crawlers are steered away from the markdown mirrors only. Each //index.md # is a full-text copy of the HTML page beside it (generated by # tools/gen_seo_artifacts.py), so letting it into a search index would put two URLs # with the same content against each other. The HTML is the canonical version and is # what should rank. A crawler obeys only the most specific group that names it, so the # Allow line is repeated here rather than inherited. User-agent: Googlebot Allow: / Disallow: /*.md$ User-agent: Bingbot Allow: / Disallow: /*.md$ # AI crawlers and answer engines are welcome, including the markdown mirrors, and are # named explicitly so the intent is on the record rather than inferred from the # wildcard above. Note that Google-Extended and Applebot-Extended are not crawlers: # they govern whether content may be used for AI training and answer grounding. Allowing # them is a deliberate opt-in and does not affect classic search ranking either way. User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / Sitemap: https://latchel.com/sitemap.xml # Curated index for language models: https://latchel.com/llms.txt # Full text, including every FAQ on the site: https://latchel.com/llms-full.txt