Robots.txt is a very important file when you are looking for better search engine traffic because bots visit your websites and collect complete information & data and keep a copy of your webpages in their servers. In order to allow access to your webpages you need to generate a Robots.txt file which allows/disallows bots access to files/directories on your directories.
Some Robots.txt Templates for easy use!
Allow All Bots 
recommended)
User-Agent: *
Allow: /
Block all robots
User-Agent: *
Disallow: /
The following example “/robots.txt” file specifies that no robots should visit any URL starting with “/cyberworld/map/” or “/tmp/”, or /foo.html:
# robots.txt for http://www.example.com/
User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
This example “/robots.txt” file specifies that no robots should visit any URL starting with “/cyberworld/map/”, except the robot called “cybermapper”:
# robots.txt for http://www.example.com/
User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:
This example indicates that no robots should visit this site further:
# go away
User-agent: *
Disallow: /
Just add the above codes in a text file and save it as robots.txt and upload to your root folder of your hosting account using your ftp and its done. You have allowed/disallowed all the bots using the above file.
Nice post for setting up the Robots.txt file. Would greatly help newbies
[Reply]
OMG
There are so many things you can do for your site,, i never knew what seo is all about
mr. admin thanks for all this useful tips
no body likes to share this tips but you do
thats why we keep coming here
Keep it up
thanks
[Reply]
Hi, the information is great. Actually its secret. I appreciate your help, thats why I subscribed for your newsletter. Do let us know about more………
[Reply]
thank you for providing such important post for me. Now I understand more.
[Reply]
The robots.txt file is a very important file and is the key to the Search Engine Bots, so one can also try out the auto robots.txt file generator available as a service with the Google Webmaster Tools, at google.com/webmasters/ login using your gmail id and under Tools you will find Generate robots.txt
[Reply]
My blog is not getting google page rank, its 6 months old. What should I do?
[Reply]
Amit Bhawani Reply:
June 13th, 2009 at 1:34 pm
@Hari, Tell me one thing why are you concerned about the Google Toolbar Page Rank? Is it really going to help you in getting Traffic or any Money?
[Reply]
You are right. But I wanted to know what that I missing out in not getting the google page rank.
And also please give me some tips in getting Traffic to my site.
[Reply]
Amit Bhawani Reply:
June 17th, 2009 at 2:56 pm
For traffic and other questions this is not the right post to continue replying here. You need to check out our help desk and post your questions there and we will help you with the best solution http://www.amitbhawani.com/help/
[Reply]
hi, Is it possible to create a robot file for the blog spot?
[Reply]