Configure Robots.txt and Robots Header Tag in Blogger Blog

If you are using Blogger as your bloggnig platform.You might have noticed about two options like Robots.txt and Robots and Custom robots header tag under the Search Preference OPitons on Settings.This part is very important for search engine to crawl and and what no to crawl information are given in such types Robot txt.

Basically there is no difference between Robots.txt and Custom robots header tag both are used for how can search engine Crawl the website and what to not.In robots.txt we are giving command lines but in case of  Robots custom header tag just we want to tick mark the basic details.

Custom Robots.txt

1.Go to Dashboard > Settings > Search Preference > Under Crawling and Indexing
2. Click on edit button just near to custom Robots.txt
3.Type the below code into the space provided 

#robots.txt of this blog/website, thanks to ipeeworld.comUser-agent: Mediapartners-GoogleDisallow:User-agent: *Disallow: /searchDisallow: /p/*Disallow: /view/*Disallow: /?m=1Disallow: /?m=0Disallow: /*?m=1Disallow: /*?m=0Allow: /Sitemap:

 Above given is an example of a SEO friendly Custom Robots.txt, Just i'm going to share what each and every line does for us

User-agent: Mediapartners-GoogleDisallow:

This line  is only for Adsense Crawlers, If you want to show ads on your blog this line make the way for adsense to crawl your blog.If you are not a Adsense publisher it is not required to keep this line you may delete this line

User-agent: *

This is used by all crawlers expect Adsense

Disallow: /search

This line indicates Which line are not to be crawled by Search engine.This line also plays a major role in making your site SEO friendly by not crawling Labels, Because we don't want any labels or achieve to be crawled by Search Engine

Disallow: /p/*

This line is used to stop crawling your blog pages

Disallow: /view/*

Stop's crawling blog's dynamic view links, If you are using Dynamic views just remove this line

Disallow: /?m=1Disallow: /?m=0Disallow: /*?m=1Disallow: /*?m=0

This line prevents from crawling mobile redirected pages. If you don't want them you will see that your blogs link with mobile friendly links

Allow: /

If you use above said line which will restrict certain links on blog, You may add links that are free to crawl 

Sitemap: http://ipeeworldcom/sitemap.xml

It is the Sitemap of your blog, Now replace with your preferred  domain URL

Custom robots header tags

You can edit Custom Robots header tag like given below (Also refer the image)
Homepage  - all and noodp
 Archive and Search pages - noindex and noodp
Default for Post and pages - all and noodp

Also Check  Prevent Blogger From Redirecting Your Blogspot Blog To Country Specific URLs

About the author

Astro Joseph

Hey Guys, I'm Astro K Joseph, Founder and CEO of IPEE World from Kerala, India. I started this blog as a passion and it now empowering thousands of readers around the globe.Here I usually share stuff related to How To, Android, Windows, Facebook, WhatsApp, Blogging, etc.

Leave a Comment