By default each website allows the Search engines robots however if you would like to restricts the robots either to not crawl any bound directory, file or the complete website then you may want the robots.txt file in which you've got to write instructions for search engine bots.
Steps to edit robots.txt on blogger
Your Site Settings › Search preferences › Crawlers and indexing
Here you will able to see two options Custom robots.txt and Custom robots header tags.These two
options would offer you the flexibility to customise your robot.txt file.In the last post i told you about
Custom robots header tags.
options would offer you the flexibility to customise your robot.txt file.In the last post i told you about
Custom robots header tags.
Now press edit button which is present after 'Custom robots.txt' option. After pressing on edit button
you can see a message " Enable custom robots.txt content? " so press "Yes" and proceed to next step.
Now you can see text area, type the content which you want to exclude a content from crawling.
Click on Save Changes button.
you can see a message " Enable custom robots.txt content? " so press "Yes" and proceed to next step.
Click on Save Changes button.
You are done!
How to block Link from search engines?
No, you have to write the URL. For example, if your want to stop robots from crawling this URL
(www.bloggerknown.blogspot.com/p/about.html) then, in the robot.txt file you will enter this command.
User-agent: *Disallow: p/about.htmlYour robots.txt file located under your main blogspot directory as for Blogger Known its located on the following url:
http://googlemediatech.blogspot.com/robots.txtBy default blogger robots.txt contains the following content:
User-agent: Mediapartners-GoogleI know this post is not a new thing but I hope it might help many newbies to understand the importance of robots.txt.
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://bloggerknown.blogspot.com/feeds/posts/default?orderby=UPDATED
0 comments:
Post a Comment