robots.txt convention (1994)
Webmasters began placing a plain-text file at their site root telling automated crawlers which paths to avoid. It became the near-universal machine channel for directing search-engine bots.
Crawlers from every major search engine learned to read the convention within a few years.
It set the pattern for sites issuing machine-readable instructions to bots, later copied by llms.txt and agents.md files.
Today's x-llms-txt headers do the same job for AI agents. The difference: crawlers treated robots.txt as policy, while agents treat the pointer's destination as new, untrusted content.
