Peter Cooper's ideas about his college in Limestone Springs were still taking shape when he visited the property in 1878. Before the trip, he said to his New York-based adviser, Rossiter Raymond, that there must be "no objection on the score of color in the admission." He was inheriting a girls' school that dated back to 1845, but there was potential to do more.

Cooper was approached by a group of Methodists from New York who wanted to use one of the buildings to house an orphanage and farm school. This did not work out. Also, in his correspondence with local adviser Thomas Bomar. an idea emerged of starting a school for the children of freed slaves. Education regardless of race was something that Peter Cooper had held dear to his heart since the establishment of New York'sCooper Union, in 1859.

In addition to offering a basic education, he could see that the tea and hemp plantations on the school grounds could teach pupils farm skills. Hemp rope was in high demand by the navy, and Peter thought the children of freed slaves, already strong from farm work, could help turn the raw hemp into rope. This way the schoolwould pay for itself.