![\"Robots.txt](\"https:\/\/i0.wp.com\/www.amitbhawani.com\/Images\/R\/robots-txt.png\")
Some Robots.txt Templates for easy use!<\/p>\nRobots.txt is a very important file when you are looking for better search engine traffic because bots visit your websites and collect complete information & data and keep a copy of your webpages in their servers. In order to allow access to your webpages you need to generate a Robots.txt file which allows\/disallows bots access to files\/directories on your directories.<\/p>\n
Some Robots.txt Templates for easy use!<\/p>\n
Allow All Bots :(recommended)<\/strong> Block all robots<\/strong> The following example “\/robots.txt” file specifies that no robots should visit any URL starting with “\/cyberworld\/map\/” or “\/tmp\/”, or \/foo.html:<\/strong> This example “\/robots.txt” file specifies that no robots should visit any URL starting with “\/cyberworld\/map\/”, except the robot called “cybermapper”:<\/strong> This example indicates that no robots should visit this site further:<\/strong> Just add the above codes in a text file and save it as robots.txt<\/strong> and upload to your root folder of your hosting account using your ftp and its done. You have allowed\/disallowed all the bots using the above file.<\/p>\n","protected":false},"excerpt":{"rendered":" Robots.txt is a very important file when you are looking for better search engine traffic because bots visit your websites and collect complete information & data and keep a copy of your webpages in their servers. In order to allow access to your webpages you need to generate a Robots.txt file which allows\/disallows bots access […]<\/p>\n","protected":false},"author":1,"featured_media":18492,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_genesis_hide_title":false,"_genesis_hide_breadcrumbs":false,"_genesis_hide_singular_image":false,"_genesis_hide_footer_widgets":false,"_genesis_custom_body_class":"","_genesis_custom_post_class":"","_genesis_layout":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[660],"tags":[],"class_list":{"0":"post-1109","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-seo-tips","8":"entry"},"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.amitbhawani.com\/blog\/wp-content\/uploads\/2010\/12\/robots-txt.png?fit=126%2C63&ssl=1","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2919,"url":"https:\/\/www.amitbhawani.com\/blog\/remove-website-pages-index\/","url_meta":{"origin":1109,"position":0},"title":"Remove\/Block Website Pages from Google\/Yahoo\/MSN Index","author":"Amit Bhawani","date":"January 5, 2009","format":false,"excerpt":"You are just starting a new website and some of your pages are already indexed by google? Now you want Google to deindex these pages and you are confused how to get this done? Well, the answer is quite easy, the way google offers you different options to index your\u2026","rel":"","context":"In "SEO Blog"","block_context":{"text":"SEO Blog","link":"https:\/\/www.amitbhawani.com\/blog\/category\/seo-tips\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.amitbhawani.com\/blog\/wp-content\/uploads\/2010\/12\/No-Access-SE-Robots.gif?fit=380%2C381&ssl=1&resize=350%2C200","width":350,"height":200},"classes":[]},{"id":3488,"url":"https:\/\/www.amitbhawani.com\/blog\/get-lost-google-rankings\/","url_meta":{"origin":1109,"position":1},"title":"How to Get Google Rankings Back with Reconsideration","author":"Amit Bhawani","date":"March 5, 2009","format":false,"excerpt":"If your website\/blog is totally banned in Google search engine because of some unknown reasons you need to signup with Google Webmaster Tools, add your website in your account and verify it. Here are some of the reasons why your website may have got banned : Hidden Links or Text\u2026","rel":"","context":"In "Google"","block_context":{"text":"Google","link":"https:\/\/www.amitbhawani.com\/blog\/category\/google\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":9298,"url":"https:\/\/www.amitbhawani.com\/blog\/url-removal-request-google-reinclusion-tips\/","url_meta":{"origin":1109,"position":2},"title":"URL Removal Request from Google & Reinclusion Tips","author":"Amit Bhawani","date":"February 17, 2011","format":false,"excerpt":"I have always been recommending webmasters to sign up with Google Webmaster Tools, here is an excellent reason for the same. Google offers with an option to Remove URL's from the search engine and this is available directly to the public. This means that anyone can report any web page\u2026","rel":"","context":"In "Google"","block_context":{"text":"Google","link":"https:\/\/www.amitbhawani.com\/blog\/category\/google\/"},"img":{"alt_text":"Remove cancel","src":"https:\/\/i0.wp.com\/www.amitbhawani.com\/blog\/wp-content\/uploads\/2011\/02\/Remove-150x150.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":4578,"url":"https:\/\/www.amitbhawani.com\/blog\/bing-seo-ranking-guide\/","url_meta":{"origin":1109,"position":3},"title":"SEO for Bing – Top Ranking Tips & Tools","author":"Amit Bhawani","date":"June 5, 2009","format":false,"excerpt":"I was recently having a discussion with chetan bhawani who informed me that his thewwwblog received quite good number of visitors from the search engine Bing.com which was microsofts new launched venture after he checked the bing traffic reports. Then we knew there would be many people who would be\u2026","rel":"","context":"In "Bing Blog"","block_context":{"text":"Bing Blog","link":"https:\/\/www.amitbhawani.com\/blog\/category\/bing\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.amitbhawani.com\/blog\/wp-content\/uploads\/2010\/12\/Bing-SEO-Logo.png?fit=600%2C471&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.amitbhawani.com\/blog\/wp-content\/uploads\/2010\/12\/Bing-SEO-Logo.png?fit=600%2C471&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.amitbhawani.com\/blog\/wp-content\/uploads\/2010\/12\/Bing-SEO-Logo.png?fit=600%2C471&ssl=1&resize=525%2C300 1.5x"},"classes":[]},{"id":1119,"url":"https:\/\/www.amitbhawani.com\/blog\/tools-create-google-sitemap-submit\/","url_meta":{"origin":1109,"position":4},"title":"How to Create a Google Sitemap & Submit It","author":"Amit Bhawani","date":"August 20, 2008","format":false,"excerpt":"If you are a website owner you would obviously expect your website to gain visitors from search engines. Once you setup your robots.txt file you need to get started with setting up a sitemap of your complete website which can be submitted to google search engine and using this sitemap\u2026","rel":"","context":"In "SEO Blog"","block_context":{"text":"SEO Blog","link":"https:\/\/www.amitbhawani.com\/blog\/category\/seo-tips\/"},"img":{"alt_text":"Free Sitemap Generator Tools","src":"https:\/\/i0.wp.com\/www.amitbhawani.com\/Images\/S\/Sitemap-Generation-Tool.PNG?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":400,"url":"https:\/\/www.amitbhawani.com\/blog\/finding-when-google-spiders-last-visited-your-website\/","url_meta":{"origin":1109,"position":5},"title":"Finding when Google Spiders last visited your website","author":"Amit Bhawani","date":"April 7, 2008","format":false,"excerpt":"Google bots keep visiting your website if your pages are search engine friendly, you have added a robots.txt file, have optimised your website and have your website updated regularly with fresh original content. Recently i recieved a email from one of my visitor who was interested to know when was\u2026","rel":"","context":"In "Google"","block_context":{"text":"Google","link":"https:\/\/www.amitbhawani.com\/blog\/category\/google\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.amitbhawani.com\/blog\/wp-content\/uploads\/2010\/12\/Google-Cache-Results.jpg?fit=631%2C492&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.amitbhawani.com\/blog\/wp-content\/uploads\/2010\/12\/Google-Cache-Results.jpg?fit=631%2C492&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.amitbhawani.com\/blog\/wp-content\/uploads\/2010\/12\/Google-Cache-Results.jpg?fit=631%2C492&ssl=1&resize=525%2C300 1.5x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.amitbhawani.com\/blog\/wp-json\/wp\/v2\/posts\/1109","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.amitbhawani.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.amitbhawani.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.amitbhawani.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.amitbhawani.com\/blog\/wp-json\/wp\/v2\/comments?post=1109"}],"version-history":[{"count":0,"href":"https:\/\/www.amitbhawani.com\/blog\/wp-json\/wp\/v2\/posts\/1109\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.amitbhawani.com\/blog\/wp-json\/wp\/v2\/media\/18492"}],"wp:attachment":[{"href":"https:\/\/www.amitbhawani.com\/blog\/wp-json\/wp\/v2\/media?parent=1109"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.amitbhawani.com\/blog\/wp-json\/wp\/v2\/categories?post=1109"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.amitbhawani.com\/blog\/wp-json\/wp\/v2\/tags?post=1109"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nUser-Agent: *
\nAllow: \/<\/p>\n
\nUser-Agent: *
\nDisallow: \/<\/p>\n
\n# robots.txt for http:\/\/www.example.com\/
\nUser-agent: *
\nDisallow: \/cyberworld\/map\/ # This is an infinite virtual URL space
\nDisallow: \/tmp\/ # these will soon disappear
\nDisallow: \/foo.html<\/p>\n
\n# robots.txt for http:\/\/www.example.com\/
\nUser-agent: *
\nDisallow: \/cyberworld\/map\/ # This is an infinite virtual URL space
\n# Cybermapper knows where to go.
\nUser-agent: cybermapper
\nDisallow:<\/p>\n
\n# go away
\nUser-agent: *
\nDisallow: \/<\/p>\n