At 07:49 AM 8/4/2011, Jim Loudon wrote: > > I'm not sure about Freepages but I know in many areas of RootsWeb staff > > monitors which spiders are allowed access to the sites. > >Are you a Rootsweb official spokesperson? 1) As others have said, a robots.txt file in your individual account is useless. To have any effect at all, it must be in the root directory and because of the configuration of the Freepages server, users do not have access to their root directory. 2) The bot must be well mannered and follow the instructions in any robots.txt file. The bots who harvest email addresses are NOT well mannered. They will ignore robots.txt 3) Staff has always monitored server access and been very particular about which bots (spiders) are allowed. Pat