Joseph Goodhue was a farmer, physician, and military surgeon in the Portsmouth, New Hampshire, area before retiring with his wife to Deerfield, Massachusetts. Their daughter Harriet (1799-1874) attended Deerfield Academy and married Dr. Stephen West Williams (1790-1855) of Deerfield in 1818. Stephen bought a house in 1822, and Dr. and Mrs. Goodhue lived in the north portion. They enlarged the house, making it two rooms deep on both the first and second floors, and in 1824, Stephen sold the north half to Dr. Goodhue.