What is Robots.txt in SEO?

Robots.txt is a protocol designed for Google Bot/web robots to know which page to crawl and which Page not to crawl. It’s a text file to instructs crawlers or robots how to index and crawl the website pages.

Is robots txt necessary for SEO?

Usually, Google finds pages and index them. It’s not mandatory to have the robots.txt file for all the websites. If robots find duplicate pages or not important pages, they automatically non-index those pages.

What is allow and disallow in robots txt?

Allow and Disallow in robots.txt determines robots or crawler to crawl complete website or do not crawl complete website.

Allow example:

User-agent: *

Allow: /

Means, the entire website will be crawled and indexed

Disallow example:

User-agent: *


Means, the entire website will not be crawled and indexed

User can also use a combination of Allow & Disallow

If you have a non-public pages or preventing indexing of resources that you don’t want to be indexed, then you can instruct robots.txt as below

User-agent: *

Disallow: /seo-services

Disallow: /termsandconditions

disallow: /ajax

How do I find the robots txt for a website?

Robots.txt can find in multiple methods. Anyone can use Tools like or to find the robots.txt file

You can enter url/robots.txt (

Latest Blogs

Free Demo


Contact Phone
+91 98458 34306
Office Address
347, HMT Main Road, Mathikere, Bangalore
Office hours
Mon - Sun: 9:00am - 8:00pm
Social media & sharing icons powered by UltimatelySocial
Book Your Demo

Up-Skill your career with one of the leading Selenium and Digital Marketing training company in Bangalore, India