Throughout the war Charlotte remained with the Collins family, moving between Somerset Place and Hurry Scurry. When the children were grown she was given the position of housekeeper. Somerset Place had been her home for over half her life, and she chose to remain with the family whose heartbreak had been her own. She died at Somerset Place during the years following the war.

It was quite some time after the Civil War, when the house had been closed up and long-neglected, that the first of the stories of Charlotte's ghost surfaced. Some little boys who had been fishing at Lake Phelps overstayed their time and were caught by darkness. While taking a shortcut through the backyard of the Collins mansion they were surprised to see the faint glimmer of a light at one of the windows on the second floor. It disappeared as they watched, only to reappear in the window of Charlotte's third-floor bedroom. A shadowy figure moved back and forth in front of the tiny oblong window as the mesmerized children watched. After a few minutes the light was extinguished. The young fishermen had seen enough for one night and hurried home as though pursued by the devil himself.

Soon other residents began reporting mysterious lights at Somerset Place. Some witnesses saw the light of a single flickering candle that appeared to be held by someone who was pacing up and down the second floor hallway. The light would come to a stop in front of one of the bedrooms Then it would linger a few moments before continuing to the door of the next bedroom--back and forth, back and forth. Rumor had it that it was Charlotte traveling repeatedly between the two bedrooms where Hugh and Edward had been brought after their deaths.

In later years other reports told of not one candle, but three. The light in the middle, held higher than the others, was thought be belong to Charlotte, while those on either side were believed to be held by the ghosts of Hugh and Edward.

There are those who will tell you that they have heard sobbing and crying coming from a certain room on the third floor. It is in that room that the plaster will not stay on the chimney wall. The story goes that the plaster keeps falling because the body of Charlotte's baby is buried in the wall, though no one knows who fathered the child or when it was born or how it came to be laid to rest in a wall. Neither is it known for certain whether the cries are those of a mother or a newborn infant. Perhaps they are both.