Joseph Emerson Brown (1821-1894) was born to Mackey Brown (1797-1874) and Sally Rice (1797-1874) in South Carolina. In 1844, he opened an academy in Canton, Georgia and began reading law. He attended law school at Yale for one year, where he earned his degree in 1846. Afterwards, he returned to Canton to practice law. In 1847, he married Elizabeth Grisham Brown (1826-1896) and they had eight children: Julius Lewis Brown (1848-1910), Mary Virginia Brown (1850-1927), Joseph Mackey Brown (1851-1932), Franklin Pierce Brown (1853-1871), Elijah Alexander Brown (1857-1926), Charles McDonald Brown (1859-1881), Sally Eugenia Brown (1862-1942), and George Marion Brown (1865-1942). Joseph Emerson Brown served as Georgia State Senator for the 41st District from 1849 to 1850; Georgia Presidential Elector in 1852; Georgia Superior Court Judge for the Blue Ridge Circuit from 1855 to 1857; Trustee of the University of Georgia from 1857-1889; Georgia's governor from 1857 to 1865; Chief Justice of the Georgia Supreme Court from 1868 to 1870; a board member on the Atlanta Board of Education from 1869 to 1889; president of the Western and Atlantic Railroad from 1870 to 1890; and U.S. Senator from 1880 to 1891. He is buried in Atlanta's Oakland Cemetery.

Mary Virginia Brown (1850-1927) attended the Lucy Cobb Institute in Athens, the Georgia Baptist College in Madison, and the Atlanta Classical College. She was a member of the Daughters of the American Revolution, Colonial Dames, and the United Daughters of the Confederacy. In 1874, Brown married Dr. Elijah Lewis Connally (1837-1930), and they had six children: Joseph Brown Connally (1875-1920), Mary Temperance Connally (1877-1953), Elizabeth Grisham Connally (1878-1880), Thomas Whipple Connally (1881-1927), Sally Brown Connally (1884-1958), and Frances Connally (1888-1941). Joseph Brown Connally attended Emory College from 1891 to 1894 and graduated from the University of Georgia in 1896. He was associated with Penn Mutual Life Insurance Company and then went into the automobile business with E. H. Inman. In 1915, he became general manager of Connally Realty Company. He directed the construction of the Connally Building, an office building located at the corner of Alabama and Whitehall Streets, and he was owner of the Connally Barber Shop. In 1917, he completed officers training camp at Fort McPherson and received a commission as Captain. He was assigned to Camp Gordon and took command of the 325th Infantry, 163rd Brigade, 82nd Division. He became the regimental unit supply officer. In October 1918, he was severely gassed in the Argonne Forest offensive. He was hospitalized for his injuries and later released. In December 1918, he was appointed regimental intelligence officer and continued in this capacity until April 1919. Afterwards, he returned to Atlanta where he was once again hospitalized for the injuries he had received. He died at Fort McPherson Hospital on August 26, 1920 and was buried at Oakland Cemetery.

Mary Temperance Connally (1877-1953) attended the West End Academy and Washington Seminary in Atlanta and later served as the first president of the Washington Seminary Alumnae. She was a trustee of the Andrew and Frances Stewart Baptist Good Will Center, Bessie Tift College, and Spelman College. She was involved with the Southern Baptist Historical Society of the Southern Baptist Convention, the Atlanta Women's Organization for the Council of Defense during World War I, the Atlanta Historical Society, the Daughters of the American Revolution, the United Daughters of the Confederacy, the Atlanta Women's Club, the Atlanta Art Association, the League of Women Voters, the Service Star Legion, and the YWCA. Connally married John Schaffner Spalding (1875-1943) in 1902, and they had five daughters: Mary Brown Spalding (1903–1989), Constance Schaffner Spalding (1906–1997), Frances Connally Spalding (1908–1996), Sally C. Spalding (1910-1988), and Elizabeth Gresham Spalding (1913–1971). Constance Schaffner Spalding married Albert Sidney Anderson Jr. (1904-1988) in 1931, and they had two children: Constance S. Anderson (1935- ) and Albert Sidney Anderson III (1940- ).