R – Google indexed the test folders on the website :( How to restrict the web crawlers!


Help Help! Google indexed a test folder on my website which no one save I was supposed to know about :(! How do I restrict google from indexing links and certain folders.

Best Solution

Use a robot exclusion file, or better yet password protect your test areas! Using a robots.txt file to "protect" areas you don't want others to see is a little like hanging a sign on your back door saying "I've left this open but please don't come in" :)

If you sign up for Google webmaster tools, you can request removal of a search result if you ensure it's no longer accessible by their crawler.

Related Question