Glossary term
Robots.txt
Robots.txt is a file that gives supported crawlers instructions about fetching paths on a site.
It is not an access-control system. Blocking a fetch does not necessarily prevent an address from appearing in search, and can hide page-level indexing instructions.
Does robots.txt keep a page private?
No. It gives crawling instructions to supported bots. Protect private information with appropriate authentication and access controls.


