# a sample robots exclusion file that excludes all # but the most significant robots from visiting your site # the default Disallow rule permits all of your web to be # crawled by the allowed agents, you will want to change this # see http://info.webcrawler.com/mak/projects/robots/exclusion.html # for further details # Note that the blank lines between sections are significant User-agent: Lycos Disallow: User-agent: Googlebot Disallow: # tell all other robots to go away User-agent: * Disallow: /