# allinon21.win
#
# Everything here is free to read and there is nothing to protect: no accounts,
# no user data, no paywall, nothing to sell. Search engines are welcome to
# crawl every page, and nothing in this file is meant to make the site harder
# to find.
#
# What this file can and cannot do, stated first, because the rest only makes
# sense once it is clear:
#
#   robots.txt is advisory. RFC 9309 §1 says these rules "are not a form of
#   access authorization"; §3 says the protocol "is not a substitute for valid
#   content security measures". A crawler that ignores this file is not
#   breaking anything it was obliged to obey, and operators are documented
#   doing exactly that — Apple says plainly that its iTMS agent "does not
#   follow robots.txt", and Cloudflare reported in August 2025 that Perplexity
#   reached content through undeclared crawlers after being disallowed.
#   This site is public, static and unauthenticated. Bulk extraction can be
#   discouraged here; it cannot be prevented here. Preventing it would take
#   blocking at the CDN, which this site does not do.
#
#   robots.txt also has no field for "a short description only". It is a
#   crawl / do-not-crawl switch and nothing else. The length limit is set with
#   the max-snippet directive, served as an X-Robots-Tag response header from
#   vercel.json, and Google is explicit that such settings "can be read and
#   followed only if crawlers are allowed to access the pages that include
#   these settings". That is why the crawlers that quote the site are not
#   blocked below: blocking them would throw away the one control that
#   actually shortens what they quote.
#
# Why there is a line to draw at all. Every strategy figure here is
# conditional. The index numbers are Hi-Lo, six decks, stand-on-soft-17. The
# basic strategy charts are generated live from the rules the reader selects,
# so there is no single correct chart to copy. A figure reproduced without its
# conditions is not a shorter version of the advice, it is different advice,
# and people may risk money on it. Find the site, quote a sentence, link back.
# Lifting the tables into a corpus is the case this file argues against.


# ---------------------------------------------------------------------------
# Everyone
# ---------------------------------------------------------------------------

# Content signals, per Cloudflare's Content Signals Policy. The definitions
# below are theirs, quoted so the terms mean what their author says they mean:
#
#   search:   building a search index and providing search results (e.g. returning
#             hyperlinks and short excerpts). Search does not include providing
#             AI-generated search summaries.
#   ai-input: inputting content into one or more AI models (e.g. retrieval
#             augmented generation, grounding, or other real-time taking of
#             content for generative AI search answers).
#   ai-train: training or fine-tuning AI models.
#
#   (a) If a content-signal = yes, you may collect content for the corresponding use.
#   (b) If a content-signal = no, you may not collect content for the corresponding use.
#   (c) If the website operator does not include a content signal for a corresponding
#       use, the website operator neither grants nor restricts permission via content
#       signal with respect to the corresponding use.
#
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
#
# Stated plainly, because this file does not claim things work when they do not:
# no crawler operator is documented as honouring this field. Cloudflare says
# content signals "express preferences; they are not technical countermeasures
# against scraping" and that "Some companies might simply ignore them." Google
# supports four robots.txt fields and ignores the rest, so this line is inert
# there. The IETF draft that would have standardised it expired unadopted.
#
# It is here for the one part that is not decoration: the Article 4 reservation
# above, which is a legal statement rather than a request, and which is made in
# full at /tdm-policy along with the conditions attached to it. See also the
# TDMRep reservation in /.well-known/tdmrep.json, the tdm-reservation and
# tdm-policy response headers, and the meta tags on every page.

User-agent: *
Content-signal: search=yes, ai-input=no, ai-train=no
Allow: /

# /offline and /404 used to carry Disallow lines here. They no longer do, and
# that is a correction rather than an omission. Google documents that a URL
# blocked in robots.txt "can still be indexed if linked to from other sites",
# and that a crawler which cannot fetch a page "will never see the noindex
# rule" — so the Disallow was suppressing the one directive that would have
# worked. Both pages are now served with X-Robots-Tag: noindex from
# vercel.json, which is what actually keeps "You are offline" out of a results
# page, and it works only because crawling them is allowed. Neither is in the
# sitemap.


# ---------------------------------------------------------------------------
# Collection in bulk, rather than to help anyone find the site
#
# Declined. Every token below is one its own operator publishes as the
# training or bulk-collection agent, and in each case that same operator
# publishes a separate token for search and citation which is left alone.
# That split is the whole design: the site stays findable and quotable, and
# stays out of the corpora.
#
# The cost is real and worth naming rather than hiding. Declining
# Google-Extended gives up grounding in Gemini; declining CCBot and Ai2Bot-Dolma
# takes the site out of two open archives that researchers use as well as model
# builders. Those are deliberate trades.
# ---------------------------------------------------------------------------

# OpenAI. "Disallowing GPTBot indicates a site's content should not be used in
# training." OAI-SearchBot, which surfaces sites in ChatGPT's search features,
# is a separate token and is not restricted.
User-agent: GPTBot
Disallow: /

# Anthropic. ClaudeBot collects "web content that could potentially contribute
# to their training". Claude-SearchBot and Claude-User are separate tokens and
# are not restricted. Anthropic states its bots honour robots.txt.
User-agent: ClaudeBot
Disallow: /

# Google. Google-Extended covers training for Gemini and grounding in Gemini
# Apps. Google documents that it "does not impact a site's inclusion in Google
# Search nor is it used as a ranking signal in Google Search", so this line
# costs nothing in Search. Googlebot is named nowhere in this file and crawls
# everything.
User-agent: Google-Extended
Disallow: /

# Apple. Applebot-Extended is not a crawler at all — it is a flag on data
# Applebot already fetched, governing use in Apple's foundation models. Apple
# documents that content "will remain discoverable through Spotlight, Siri,
# and Safari" either way.
User-agent: Applebot-Extended
Disallow: /

# Meta. meta-externalagent crawls "for use cases such as training foundation
# AI models"; meta-externalads feeds advertising products and helps nobody
# find anything. meta-webindexer, which is what gets a site cited in Meta AI
# answers, is separate and is not restricted.
User-agent: meta-externalagent
User-agent: meta-externalads
Disallow: /

# Amazon. Amazonbot content "may be used to train Amazon AI models".
# Amzn-SearchBot, which Amazon states does not crawl for training and which
# makes a site eligible to appear in Alexa answers, is separate and is not
# restricted.
User-agent: Amazonbot
Disallow: /

# Mistral. MistralAI-Training "crawls web content to help build datasets for
# training Mistral generative AI models". MistralAI-Index and MistralAI-User
# are separate and are not restricted.
User-agent: MistralAI-Training
Disallow: /

# Common Crawl. CCBot trains nothing itself, but its archive is a standard
# input to other people's training runs, so allowing it is an indirect opt-in
# to all of them at once. Note this is forward-looking only: no robots.txt
# line reaches an archive already published.
User-agent: CCBot
Disallow: /

# Diffbot. Crawls for a commercial knowledge graph sold as a dataset, which is
# the clearest case here of collection that is not discovery. Diffbot states
# robots.txt adherence is default behaviour that a commercial agreement can
# override, so this line is weaker than the others even in principle.
User-agent: Diffbot
Disallow: /

# ByteDance. Bytespider collects training data for ByteDance's own models. No
# separate search token is published for it, so unlike every other entry above
# this one has no discovery counterpart being left open — there is nothing to
# split.
User-agent: Bytespider
Disallow: /

# Allen Institute for AI. Ai2Bot-Dolma gathers the Dolma pretraining corpus,
# which is the same indirect opt-in that CCBot is: an open dataset that other
# people train on. Ai2Bot without the suffix is a separate token and is not
# restricted. Ai2 states its crawlers honour robots.txt.
User-agent: Ai2Bot-Dolma
Disallow: /

# Cohere. The token says what it collects.
User-agent: cohere-training-data-crawler
Disallow: /

# img2dataset. Not a company's crawler but a tool anyone can point at a site to
# build an image training set from it. Named for what it does.
User-agent: img2dataset
Disallow: /


# ---------------------------------------------------------------------------
# Welcome, and deliberately not written as groups
#
# The crawlers that help people find the site are wanted here: Googlebot,
# bingbot, Applebot, DuckAssistBot, OAI-SearchBot, Claude-SearchBot,
# PerplexityBot, meta-webindexer, Amzn-SearchBot, MistralAI-Index, YouBot, and
# the user-triggered fetchers that visit a page because a reader asked a
# question about it.
#
# None of them has a group below, because a group would add nothing.
# RFC 9309 §2.2.2: "If no match is found amongst the rules in a group for a
# matching user-agent or there are no rules in the group, the URI is allowed."
# Silence already permits them. And a named group is not merged with the
# "User-agent: *" group — §2.2.1 makes the wildcard a fallback used only
# when no named group matches — so a bare "Allow: /" per crawler would
# quietly exempt each one from any future site-wide rule. The previous
# version of this file had twenty-two such groups, and they were doing
# precisely that damage to its own /offline and /404 exclusions. Intent
# belongs in a comment; a group is a rule.
#
# Several of these fetch on a reader's behalf and their operators say plainly
# that robots.txt may not apply to them at all — OpenAI on ChatGPT-User:
# "Because these actions are initiated by a user, robots.txt rules may not
# apply." Perplexity and Google say the same of theirs. Nothing here would
# have controlled them in either direction.
# ---------------------------------------------------------------------------

# Applebot is the one exception, for a documented reason. Apple: "If robots
# instructions don't mention Applebot but mention Googlebot, the Apple robot
# will follow Googlebot instructions." This file names neither today, so
# nothing is misrouted — naming Applebot means a Googlebot group added here
# later cannot silently capture it.
User-agent: Applebot
Allow: /


# ---------------------------------------------------------------------------
# Three things this file does not do, and why
#
# The Content-Signal line above is a rights reservation, not a control. It is
# written knowing no operator is documented honouring the field, because the
# Article 4 sentence it carries is a legal statement that does not depend on a
# crawler parsing it. Judge it on that, not on compliance.
#
# /llms.txt is not an opt-out. It exists, it leads with the conditions above,
# and it is worth keeping for the one audience measured to read it: agentic
# coding tools. It has no effect on search, ranking or training, and its
# author's stated purpose is the opposite of this file's — making a site
# easier for models to consume. Ahrefs found 97% of published llms.txt files
# were never requested at all in May 2026. It is a courtesy note, not a
# control.
#
# /search.json is the whole site's search index in one file, and it is left
# crawlable on purpose. It carries titles and one-line summaries only — no
# index numbers, no charts — which is the same text already in each page's
# meta description. Blocking it would look like a defence without being one.
# ---------------------------------------------------------------------------

Sitemap: https://allinon21.win/sitemap-index.xml