Isaac Harris Cary was born in the North End neighborhood of Boston, Massachusetts on November 3, 1803, the seventh child of Jonathan and Mary Cary. In 1824, Isaac and his brother William formed a partnership and ran a fancy goods imports business, Isaac H. Cary & Co. on Washington Street in Boston. The brothers opened a store in New York and William moved there full-time. In 1831, Isaac married Phebe P. Pratt of Roxbury and they would have three children, two of them living to adulthood. After doing business in New York City and later in New Orleans, Isaac and his family settled in the Jamaica Plain section of Boston, purchasing large land holdings and developing real estate. One of the finest lots he owned was developed for his country estate in 1850, an Italianate/Second Empire-style mansion perched atop an outcropping of Roxbury puddingstone. The large home with a rear three-story tower remained in the Cary family under his single daughter Susanna’s ownership until her death in 1913.