Robots.txt is a text file placed in a website's root directory that provides instructions to search engine crawlers about which pages or sections of the…