Ive been playing with my google webmaster tools and forun that some of my diy dating sites have the following in the robots.txt file
User-agent: *
Disallow:

This means that those sites wont be crawled by the search engines.

Any reason for this and is there an ability for us to alter this.

Also why is it on some of my didy dating sites and not others? Is this something to do with which domain I choose to add a subdomain on?

The robots.txt file exists for all sites and tells search engines to spider your whole site.

Gary

Sorry I was confusing:


Which excludes all robots from the entire server.

With:



Which allows all robots complete access, same as a blank robots.txt file

Hi, as Gary mentioned our robots.txt file is set to

User-agent: *
Disallow:

which tells search engines to spider your whole site. This file exists across all of the sites and is not somethings that you should really need to alter as there are no areas of your site that you would need to restrict from being spidered by a search engine.