The Burrell Baronets of Knepp

The lands of Drummond were the property of the Drummond family from the 14th century, and the original tower house was built over several years by John Drummond, 1st Lord Drummond of Cargill, from about 1490. In 1605 the 4th Lord Drummond was created Earl of Perth, and added to the castle. John Drummond, 2nd Earl of Perth, laid out the first terraced garden around the castle in the 1630s.

The castle was sacked by the army of Oliver Cromwell in 1653, during the Wars of the Three Kingdoms. James Drummond, 4th Earl of Perth was Lord Chancellor of Scotland under King James VII. He began the mansion house in 1689, before being imprisoned following the deposition of King James by William of Orange. He later fled to the exiled Jacobite court in France. The Drummonds continued to support the Jacobite cause in the uprisings of 1715 and 1745. The family retained control of the estate until 1750 when the Drummond properties were declared forfeit and seized by the state. The estate was managed by the Commissioners for Forfeited Estates until 1784, when it was sold to Captain James Drummond. He began a number of improvements that were continued by his daughter Sarah and her husband Peter Drummond-Burrell, 22nd Baron Willoughby de Eresby (1782–1865). These included the formal gardens and terraces in the 1830s. Queen Victoria visited the gardens in 1842.

Drummond Castle passed to Clementina Drummond-Willoughby, 24th Baroness Willoughby de Eresby (1809–1888), and then to her son, Gilbert Heathcote-Drummond-Willoughby, 1st Earl of Ancaster (1830–1910). The upper stories of the tower house were rebuilt and heightened in pseudo-medieval style in 1842–53. The mansion was renovated in 1878, to designs by George Turnbull Ewing. The 3rd Earl of Ancaster and his wife, Nancy Astor, replanted the gardens in the 1950s. The castle is now the seat of the 28th Baroness Willoughby de Eresby.