Brave’s help page ties its unnamed crawler to Googlebot’s rules

Search Engine Journal spelled out the mechanics for site owners on 6 November. Robots.txt, the file that tells crawlers to stay out, works by name, and Brave’s crawler advertises none. Anything barred to Googlebot is barred to Brave’s crawler too, Brave’s help page says, and it gives one reason for having no separate name: some sites admit only Google.

That leaves Googlebot and Brave’s crawler sharing one door, open to both or shut to both. Brave’s page offers a noindex tag, a line in a page’s code asking search engines to leave it out, and Google honours that tag as well. You can see your own exposure: Search Engine Journal suggests typing site: and your domain into search.brave.com, because those results come from the index on sale to AI firms.

Search Engine Journal leaves open whether Claude uses Brave today. Web search is on Claude’s $0 Free plan on our price sheet, our 6 November check, and that plan names no search supplier.

For a live page, noindex is the listed route out and it also removes that page from Google, so the price of leaving an index sold to AI is that page’s Google traffic.