-----Original Message----- From: Brian Leverich <leverich@rootsweb.com> To: listowners-announce@rootsweb.com <listowners-announce@rootsweb.com>; rootsweb-help@rootsweb.com <rootsweb-help@rootsweb.com>; STATE-COORD-L@rootsweb.com <STATE-COORD-L@rootsweb.com> Cc: Dr. Brian Leverich <leverich@rootsweb.com> Date: Monday, March 09, 1998 7:14 PM Subject: Serious Problems Out on the Net (and at home) > > >As many of you have undoubtably noticed, the last week has been a >mess out on The Net. > >Starting one week ago, some clown has been strobing all the hosts >and routers on the Internet with an attack that is fatal (locks the >machine such that it has to be power cycled to restart) to Win 95 >and Win NT boxen. > >While this didn't directly affect RootsWeb (we're wall-to-wall Unix >servers), it did kill one of our network neighbors' NT server. >Unfortunately that NT server was providing reverse DNS for some of >RootsWeb's boxes, so the NT server's repeated crashes caused major >problems for us. > >It especially slowed mail deliveries from our list servers. > >To deal with this problem, the NT server has been "patched" to >reduce its vulnerability to attack and we have taken various other >technical measures to make us safer. > >Last night at 2am Sprintlink, with no warning to any of its >customers, attempted to upgrade the operating systems on its >backbone routers throughout the country. Things went badly south, >and the Net is still very much crippled right now. > >The Sprintlink disaster essentially disconnected pieces of RootsWeb >from The Net for several hours until we could adjust our routers to >work around the down pieces of Sprintlink's backbone. > >As the Sprintlink disaster rippled through The Net, it also took >down the routers at our feed from CRL. That happened while Karen >and I were away from our keyboards, and the CRL down wounded our >main list server. > >Some mail was lost, and we suspect some digests were damaged. We'll >be fixing things for a few days. ): > >Ultimately there is no way that an individual site can insulate >itself from a mess like last night, but we will be able to better >protect ourselves as RootsWeb grows and operates more T1 connections >to more Internet backbone carriers. > >Finally, this morning at 10am the textbase harddrive in the search >engine box glitched and wedged, killing that server. Karen and I >were working at that machine's console to revive it when the CRL >link went down, which was why we couldn't save the mail server. > >Disk problems like this should go away as we upgrade all our servers >to using redundant RAID-5 disk arrays, rather than depending on >single drives. > > *sigh* > >Thank goodness weeks like this don't happen too often. As noted >above, we'll be doing what we can to insulate ourselves from these >sorts of problems in the future. > >One other thought: Karen and Brian will be away from our consoles all >day tomorrow traveling on business that is critical to RootsWeb's >future. It worries us a lot to leave the servers without someone >physically at the consoles, but there will be some great sysadmins >monitoring the site remotely and we think the potential benefits of >this trip outweigh the risks of leaving the servers for a day. > >We apologize in advance if anything goes wrong, and we will be back >late tomorrow night in any case. Cheers, B. > > >-- >Dr. Brian Leverich Co-moderator, soc.genealogy.methods/GENMTD-L >RootsWeb Genealogical Data Cooperative http://www.rootsweb.com/ >P.O. Box 6798, Frazier Park, CA 93222-6798 leverich@rootsweb.com > >