Half a mile east of Rosehearty, a ruined castle sits on the slopes of a hill overlooking a sandy bay. Pitsligo Castle was originally designed as a keep dating to 1424. It was then modified in the 1570s by the Forbes of Druminnor. Despite not having a particularly strong or strategically important location, Pitsligo Castle overlooked the shipping route along the Moray Firth coast.

Scottish historian W. Douglas Simpson described Pitsligo Castle as one of the nine castles of the Knuckle, a term used to refer to the rocky headland of northeast Aberdeenshire.

The castle has an arched gateway in the outer courtyard's west wall, which is marked both with the date 1656 and the coats of arms of both the Forbes and the Eskine families. In the inner court, however, the date is shown as 1663. Several armorial plaques can be found on other inner walls, including the Pitsligo Arms of 1665 and the Royal Arms dated 1577 and 1603.

There is a tall flanking drum tower in the northeast corner of the courtyard. Pitsligo Castle's main tower had three vaulted stories, but since its abandonment nearly everything above the lowest level has disappeared.

The Forbes family made the mistake of supporting the Jacobite cause, and in 1745 their estate forfeited to the crown. Flemish troops led by William Lascelles, looted and destroyed Pitsligo Castle in 1845. Eventually, the roof was removed and the castle fell into ruin.

At some point, it appears that restoration work was begun on the castle. Some of the fallen masonry has been collected into large piles of rubble that litter the inner courtyard. Work currently appears halted.