Among the most prominent residents of 19th century Wallingford were members of the Munson family, who established a number of farms in the fertile and scenic Otter Creek Valley. The home built by Israel Munson and his bride Matilda Clark Munson circa 1845 became the centerpiece of all the Munson farms, with its Greek Revival architecture and fabulous landmark barn.

The Munsons first came to Wallingford in 1814, when wealthy Boston physician and merchant Israel Munson purchased a 250 acre sheep farm from brothers Edmond and Gilbert Douglas for his brother Isaac. Isreal purchased a Merino ram at a cost of $150, and sent the animal back to Vermont with the Douglases to breed the flock.

In March of 1815, Isaac built a barn and moved and repaired the house. About 15 years later, he had a new brick federal style house built to the south of the original house, where he and his wife lived with their 8 children. After Isaac's death, the property was divided among his sons, who also built fine farmhouses in the valley. When Isaac's brother Israel died in Boston in 1844, he left a legacy of over $40,000 to each of Isaac's eight children. One of them, Israel's namesake, married Matilda Clark the next year and hired famed architect Asher Benjamin of Boston to design and build the 2-1/2 story post and beam house around the older farmhouse that stood on the same site. Today this is the property known as White Rocks Inn.

The Munson family was influential in the local community, with one of the brothers serving in the Vermont legislature. Israel Munson was the most financially astute, becoming the local banker for the farmers in the area. The safe that he used for his banking transactions has the name "Munson" painted on the door, and still sits in the corner of the library at White Rocks Inn.

Israel and Matilda had two sons, Kirk and Isaac, who inherited the farm after their parents died. Neither of the brothers ever married, and they lived together on the farm, becoming successful dairy farmers. They hired prominent architect Clinton G. Smith of Middlebury to enlarge and embellish the barn and added the back wing onto the house in the latter part of the 19th century. Isaac died in 1908, and Kirk died in 1932, having lived all of his 86 years in the house.

Arthur and Maude Davison purchased the house and part of the land after Kirk's death, and the Davison family ran a commercial dairy operation until 1972, when government regulations and modernization of the industry made the family dairy farm a thing of the past.

The property was converted into a restaurant and inn in 1983, and has been a B&B since 1986. It was named to the National Register of Historic Places in 1986. The following is a quote from the records of the U.S. Department of Interior concerning the property.

"The five farms associated with the Munson family are distinguished by the essentially intact character of their Federal and Greek Revival style farmhouses. The five extant houses constitute one of the most outstanding groups of family-related farmhouses in Vermont. The Israel Munson and Goodyear Clark houses rank individually among the finest rural expression in the state of high-style Greek Revival architecture. Complementing the Israel Munson House, the farm's great multi-unit barn holds equal stature in Vermont agricultural architecture for its unifying decorative treatment applied by Clinton G. Smith, one of the state's most prominent latter 19th century architects."