Spencer Kellogg Brown was born August 17, 1842, in Belleville, New York. He was the son of New York abolitionist Orville Chester Brown (1811–1904), who moved his family to Kansas and took part in the founding of Osawatomie. In retaliation for what later came to be known as the Pottawatomie massacre, pro-slavery forces sacked Osawatomie on August 30, 1856. During the raid, Spencer's family house was burned to the ground by Border Ruffians, and he was captured and taken to Lafayette County, Missouri, where he was held prisoner for some time thought to be John Brown's son.