What Is Robots.TXT File

Robots.txt is a text file. If you don’t want to index any one or more pages, folders or posts of your website to Google or any search engine, so for we use robots.txt file.

Suppose your website has a page that has copy content or has a folder in your website that contains the personal information of the website and you don’t want Google or any search engine to send bots (crawler) on your website or folder so the bots are blocked by using robots.txt file.

All search engines have their own crawler who crawl the website, read the website and then give it ranking. Crawlers read all the things on your website such as titles, keywords, pages, other links, photos, content means that your entire website is scanned after the scan your website starts indexing Well and your website’s ranking and visitors begin to grow.

When you want to call a crawler to your website, it sets you in Sitemap, when you create a Sitemap

Create Robots txt File

     Apply this text in your robots file

User-agent: *

Disallow:

Create your robots file with this sentence. In this case, we allowed all crawlers to crawl (read) the website.

You can hide from search engine some pages if you want just like:

• Pages with duplicate content
• Pagination pages
• Dynamic product and service pages
• Account pages
• Admin pages
• Shopping cart
• Chats
• Thank-you pages

Let’s understand it in depth

1. Allow and Disallow to All

              Here we have shown two conditions. 

User-agent:*

Disallow:  

User-agent:*

  Disallow:/

In the first condition, we allowed all crawler to crawl the website. It means google or any search engine can send their crawler or bot to read the website data. Mainly robots txt file help to the sitemap to crawl the website properly.

In second condition we disallow all the crawlers of every search engine. This means that anyone search engine can’t read the website because we blocked the crawler by using the ( / ) symbol.

Here we use the (/) symbol to block the bot. we said to search engines that we don’t want to rank our website in the search engine. 

2. Disallow to Search Engine

User-agent: googlebot

Disallow: / 

In this case, we blocked to the google search engine to crawl the website. This means all the search engines can send their bots to crawl the website but the only google can’t read the full website.

If you want to block any search engine so insert that search engine name where we have given the name Googlebot. if you want to block the Bing search engine then write Bingbot at the place of Googlebot.

3. Disallow to any page

User-agent: *

Disallow: /contact us/

In this case, we blocked a particular page from the search engines. This means if you don’t want to crawl one or more pages of your website so can use this tag. 

we block that page when we copied text from another website and placed it into our website because after mention it the search engine won’t penalize our website, so we use this condition for protecting the website from Google or another search engine algorithm.

User-agent: *

   Disallow: /contact us/

   Disallow: /about us/

Here we have blocked two pages of the website. If we want to block more than one page so you can use this condition. If you have two pages and you don’t want to crawl both pages so you can block these pages from search engines.

4.  Disallow to any folder of website

User-agent: *

Disallow: /cgi-bin/

Disallow: /wp-admin/

User-agent: *

Disallow: /cgi-bin/

Disallow: /wp-admin/

 Here we blocked two folders of the website. If you want to block any folder of your website so you can use this condition.

By Robots file can protect your website pages if you copied, so you can save those pages from the Google algorithm which is copy because you can’t copy the content from another website. If you copy content from another website then protect that content, if you don’t want to hide this content from a search engine then mention there website name or author name of that content.

After selecting any one option from the above syntax, save it into txt format in the word document. Remember that when you save the file, you have to name the file as robots. It means the file name is robots.txt, check this name and format then save it.

Updoad Robots txt File In Website

      When you created robots txt file then you have to upload it into your website hosting (file manager).

Go to Cpanel then click on File Manager, open Public HTML folder, if you hosted more sites then select that website for which you have created the file. After selection upload the robots txt file.

When you uploaded robots file then you have to check it that your file is working or not, to check it, type robots.txt after the main URL of your website.

After inserting URL you will see your robots file and now you submitted it properly.