Rumor attributes these strange occurrences to the spirit of Jesse Shepard himself. The concerts he gave were not your average musical event. Jesse believed that souls of past, great musicians worked through his body when he played the piano in a trance like state. He gave his guests quite a show.

He would greet his guests at his lavish parties with his usual charm and sophistication. After he had them thoroughly fascinated and under his spell, he led them into the music room. He dimmed the lights and took his place at the piano seat. What followed left the guests completely astounded. He would begin to play the most beautiful music they had ever heard, seemingly improvising as he went along. His body would gracefully sway along with the melody. Then as if by magic, people began to hear a choir of heavenly voices, distant at first, then getting louder, as if they were in the same room. Some people reported hearing drums, tambourines and trumpets play along with the music. Then the chorus would gradually fade away, leaving the guests to wonder where it had come from in the first place. The lights would appear, and there was Jesse, dramatically taking his bows to an excited audience.

Jesse was also famous for his seances. The many spiritualists in San Diego who admired Jesse always knew where to go when they wanted to make contact with a long-lost relative. Always the showman, Jesse claimed he could also communicate with Beethoven, Julius Caesar, and William Shakespeare. Only the best for Jesse.

After Jesse left for Europe, the house was home to many different owners. Because so many of them experienced difficulties after living in the house, it was thought to be jinxed.

One owner was the vice president of an important bank. He ended up in Europe, not on vacation, but as a fugitive running from justice. Even the president of the bank could not be free from the spell of the house. He committed suicide about the same time his associate went on his unscheduled trip. The next owners found they could no longer make sound business decisions. Without explanation, they ended up bankrupt, losing the house through foreclosure. Disaster after disaster awaited the next couple. The wife reported her husband became more reckless as time went by. A giant gray cloud began to engulf the husband, no longer able to withstand the pressure, disappeared, never to be heard from again. One of the owners happened to be another spiritualist. He told of strange sounds at all hours of the night. Music coming from the pipes, tapping noises, and moaning and sobbing disrupted his sleep on a regular basis.

After these misfortunes became more public, all the old rumors that the house is haunted resurfaced. Speculation was that Jesse was back in town. Or possibly the man that hung himself in the observatory tower was now enjoying the beautiful view of the city from this high point. Shadowy forms were seen roaming the house and the neighborhood children wouldn't go near it.