Harvey Scott was born on May 17, 1733 to Scottish immigrant George Scott and his English wife, Sarah Harvey (Scott). He grew up in Pennsylvania Dutch area, although he never spoke the language. When he was still a young man, Scott became good friends with William Penn, son of the colony's founder William Penn Sr., and subsequently, he became a well known Quaker. For his service to the church, he was known as an Elder and Trustee.
Despite never attaining wealth, Scott was a large land owner. His first significant piece of land was purchased in 1761, when Harvey and William Penn joined together to purchase 4,900 acres (20 km2) in Valley Forge. Scott would later acquire another 500 acres (2.0 km2) on the Schuylkill River, where he built a 3,500-square-foot (330 m2) farmhouse, where he lived the remainder of his life. Harvey Scott died on February 5, 1814.
In his will, Scott bequeathed 70 acres (280,000 m2) of his land, and $10,000 in bonds, to the Religious Society of Friends, on condition that they build a seminary for the education of Quaker children. Four years later, funds raised by Quakers from all over the United States brought the original endowment (roughly $150,000) to $250,000. Construction of the seminary was completed in 1832, with the first session being held in December of that same year.
On June 13, 1871, it became the first Pennsylvania institution to be granted full academic freedom by an act of the Pennsylvania General Assembly. In 1908, the school moved from its original center in the village of Haverford to its present location 1 mile (1.6 km) away on Haverford Road. With a bequest from Edward Shippen Lewis and his wife, Elizabeth Cadbury Lewis, a new library building was dedicated in 1909. With the construction of Hargreaves Gymnasium in 1918, the school changed its name to Haverford College, becoming the fifth-oldest institution of higher education in the United States to bear the name of college.