Google's Search Central team published a new episode of its Non-Canonical Cabinet series, where they open the drawer of weird questions collected from event Q&As. This one is all about robots.txt and RFC 9309, the internet standard Google follows when it reads the file. The questions sound silly, but a couple of the answers affect how we write robots.txt for client sites.
Can you put ASCII art in robots.txt?
Technically yes, as long as every line starts with a hash, because crawlers ignore comments. Google's advice is to keep the art out of the actual directives. The presenters mentioned a site whose robots.txt was 97% comments and 3% rules. It did not break any parser, it just wasted bandwidth on every crawl.
What the parser does with lines it does not understand
The parser reads line by line and only knows four things, which are user-agent, allow, disallow and sitemap. Anything else gets silently dropped. So a line like "don't crawl secret page" is not read as a rule at all, and Google may well crawl that page anyway. If you have ever seen a client file with plain English instructions in it, this is the reason it never worked.
Emojis and non-English characters
A user-agent token can only contain letters and a dash, so an emoji or any non-English character in that line will not work. URL paths are a different case. Googlebot handles emojis, German umlauts and Chinese or Hindi characters in paths by encoding them, but older or smaller crawlers may not. Google's suggestion is to write the percent-encoded version in your disallow rules, so the chicken emoji becomes %F0%9F%90%94. If a crawler cannot process a raw emoji it may skip the whole disallow line and crawl what you wanted to protect.
Can robots.txt tell AI models not to train on your content?
No, and this is the part to read twice. robots.txt is an access preference protocol, so it only says crawl this or do not crawl that. It has no say in what a system does with the data after it has been crawled. To keep a specific AI crawler out, you have to name its user-agent token in the file and disallow it. A made up rule such as "no AI training" is thrown away by the parser like any other unknown line.
We had the same view when reviewing client files, but it is good to have it confirmed by Google directly. If you have not looked at your own robots.txt in a while, open it today and check three things. Remove any plain English lines that are not real directives, confirm the sitemap line points to a live URL, and make sure any non-English paths are percent-encoded. The next episode will be on 404 errors.