what is robots text file ? why use arobot text file ?

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.


                                   Before a search engine crawls your site, it will look at your robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.

Robots.txt files are useful:

• If you want search engines to ignore any duplicate pages on your website
• If you don’t want search engines to index your internal search results pages
• If you don’t want search engines to index certain areas of your website or a whole website
• If you don’t want search engines to index certain files on your website (images, PDFs, etc.)
• If you want to tell search engines where your sitemap is located.

Using robots.txt for SEO allows you to exclude the pages that you don’t want search engines to crawl. You may find numerous example on how to use robots.txt file. This is a short tutorial on how to use it to block search engine crawlers to not index the pages that you don’t want to. BTW robots.txt is The Robots Exclusion Protocol.