Bannerman Castle is widely considered haunted, with its eerie atmosphere stemming from local legends dating back centuries and ghost stories connected to its more recent history as a decaying military arsenal. Accounts of hauntings include spectral sightings, unexplained noises, and ghostly figures on the island. The island, known as Pollepel Island, had a dark reputation long before the castle was built. Native American tribes who first inhabited the region believed the island was haunted by malevolent spirits and would avoid it, especially at night. Dutch sailors adopted these superstitions and feared the island, believing it was controlled by the "Heer of Dunderberg," or "The Storm King". According to a legend adapted by Washington Irving in his 1822 story "The Storm Ship," the "goblin king" sent storms to attack ships and accepted sacrifices on the island. One theory suggests the goblin stories originated when Dutch sailors would leave their drunken comrades on the island as a prank. In this version, the disoriented sailors would mistake the island's features for ghouls. After Francis Bannerman VI purchased the island in 1900 and built his family's summer home and military surplus warehouse, a new set of ghost stories emerged from its history of misfortune. Paranormal enthusiasts and visitors have claimed to see apparitions of the Bannerman family and former workers on the island. Strange noises, such as disembodied voices, echoes, and footsteps, are often reported, adding to the desolate and ghostly feeling of the castle ruins.