The Brick Tavern was an important stopping point on the old Union Turnpike, and the original two-story brick structure was completed about the time of the turnpike construction by Paul Willard, who with his heirs, operated the inn for 25 years. In the first years of the 1800s, the Union Turnpike Company planned and built a road providing a link for travel from Boston to Albany. Realizing the possibility for an inn along the first leg of the route, Willard financed this substantial brick building for travellers to stop, eat and spend the night. The tollroad was later made free, and less people stayed at the inn. After subsequent ownership, the building started to suffer from deferred maintenance and it was sold to a local Quaker group. The Quakers modernized the building by constructing the mansard roof and updating the interior. They never occupied it, but rented it to tenants for income. After, it was a hospital, boarding house, and in WWII, as a barracks of sorts for soldiers training nearby at a military base. The building is now a house!