<<Tom Downing and I have both run into a curious factor with a number of our Downing ancestors having a middle name of "Ellsworth". Yet I find no Ellsworth surname in the lines at all. Any one else have any <middle name> Ellsworth's in their lines? Any ideas? Ray Downing>> The name Ellsworth originates with the tradition of naming a son after the first Union casuality in the Civil War. Colonel Elmer Ellsworth was killed by Confederate forces when he crossed into Virginia and tried to remove a rebel flag. Using his name honored his memory.