Basically it has to do with your ISP's quality of service. I had one ISP for a while - fed up with constant stream of spams coming through their ISP. Turned out they don't care if spiders get through, they disable all no-robots.txts on all sites sitting at their ISP servers. Switched to different ISP, since then one or two a month. And never reply directly to any such spam. David Samuelsen