What is Robots.txt in SEO?

Robots.txt is a protocol designed for Google Bot/web robots to know which page to crawl and which Page not to crawl. It’s a text file to instructs crawlers or robots how to index and crawl the website pages.

Is robots txt necessary for SEO?

Usually, Google finds pages and index them. It’s not mandatory to have the robots.txt file for all the websites. If robots find duplicate pages or not important pages, they automatically non-index those pages.

What is allow and disallow in robots txt?

Allow and Disallow in robots.txt determines robots or crawler to crawl complete website or do not crawl complete website.

Allow example:

User-agent: *

Allow: /

Means, the entire website will be crawled and indexed

Disallow example:

User-agent: *


Means, the entire website will not be crawled and indexed

User can also use a combination of Allow & Disallow

If you have a non-public pages or preventing indexing of resources that you don’t want to be indexed, then you can instruct robots.txt as below

User-agent: *

Disallow: /seo-services

Disallow: /termsandconditions

disallow: /ajax

How do I find the robots txt for a website?

Robots.txt can find in multiple methods. Anyone can use Tools like or to find the robots.txt file

You can enter url/robots.txt (