In 1066 it was recorded that the manor of Thornbury was held by Beorhtric, son of Aelfgar, although by Domesday it was in the hands of King William. The manor has changed hands many times during its history, being held by the Crown at intervals. In the C12 and C13, it was part of the earldom of Gloucester; the de Clare family was responsible for the foundation of the borough of Thornbury in 1243, to the south of the church and manor house. A major fire in 1236 destroyed the manor house, following which Henry III ordered that the Constable of St Briavels supply 20 oak trees from the Forest of Dean for its rebuilding. The house came to Hugh Audley on his marriage to Margaret de Clare in 1317, passing to Audley’s son-in-law, Ralph Stafford, in 1347. It is understood that a licence to crenellate was granted in the C14, and early-C14 and C15 financial accounts provide evidence for an extensive complex in which an inner court, entered through a central gate, gave access to a hall, orientated north to south, with kitchen offices to the west and a chapel, begun in 1340 and completed in 1435, to the east of the hall. Accounts also record an outer courtyard containing a range of service buildings.
The manor house was forfeited at the execution for treason of Henry Stafford, 2nd Duke of Buckingham, following the Rebellion of 1483, but was restored to the family and inherited by his son Edward, 3rd Duke, in 1498, who made it his principal seat. Plans were laid for the creation of an ambitious fortified house and a licence to fortify, crenellate, and embattle the manor house was granted in 1510. With the hall and chapel of the existing manor house forming the east range of an inner courtyard, Buckingham set about building an elaborate palace-castle, which demonstrated the involvement of masons of the highest quality, and was apparently modelled on Richmond Palace, at that time England’s most splendid royal residence. To complement his bold plans for the castle, Buckingham enclosed some thousand acres including not only the Newe Parke north of the castle, but two further adjacent parks, Marlwood and Eastwood, as well as a coneygar; the 1521 inventory recorded 1500 head of deer in the three parks. In 1514 the Duke obtained a licence to found a college of priests attached to the adjacent parish church of St Mary; the project was not realised.
Thornbury Castle's pleasure gardens lay to the south and south-east of the castle, surrounded by a high stone wall. In the survey made in 1521 (see below) the entire enclosure was referred to as the 'privy garden', whilst the area immediately to the south of Buckingham's principal range of apartments was described as a 'proper garden', but in a late-C16 inventory of the estate this area is identified as the privy garden, the name it retains to this day. There is both physical and documentary evidence that a two-storey timber gallery, roofed with slate, enclosed the privy garden, entered from the apartments; such galleries had featured in the gardens at Richmond. On the south side of the garden, the gallery gave access to a raised walkway crossing the adjacent churchyard, and leading directly to the ducal pew (now lost) at the east end of the church (the church is listed at Grade I). C16 tiles have been found in the north-east corner of the privy garden, and it has been suggested that these may have lined the walkways, though they are more likely to be associated with the demolished Duke of Bedford's apartments in this area. The eastern portion of the walled garden enclosure was described in the 1521 survey as being 'a goodly garden to walk in', and is known as the 'goodly garden' today; this area originally communicated with a large orchard, to the east.
Edward, Duke of Buckingham, was executed on the orders of Henry VIII following an investigation for treason in 1521 – the Duke's ostentatious behaviour and wealth, as evidenced by his lavish building programme, having exacerbated the suspicion with which he was viewed – and the estate was confiscated, remaining in Crown ownership until 1554. Henry sent surveyors to make a record of his new acquisition shortly after Buckingham’s death, and their account provides a detailed description of the castle and estate. Although works were not recommenced, the buildings were maintained and periodically used; Princess Mary visited during the 1520s, and Henry VIII and Anne Boleyn stopped on a royal progress in 1535. Thornbury Castle was restored to the Staffords in 1554 when it was granted to Buckingham’s son, Lord Henry Stafford, by Queen Mary. The upkeep of the castle proved too expensive, however, and it fell into ruin, eventually coming into the ownership of a branch of the Howard family in 1637 and remaining in their hands until the 1960s.
The east range, comprising the original medieval hall and chapel, was demolished at some point before 1732. No pictorial representations survive of the range, which is described in the late-C16 inventory; archaeological investigation has demonstrated the survival of this part of the castle as a buried feature. Although part of the castle – principally the section of the west range to the south of the gatehouse – served as lodgings and a farmhouse in the C18 and early C19, much of the building was ruinous, and it was not until the C19 that it was brought back into use as a high-status residence. In 1849 Henry Howard commissioned Anthony Salvin to restore the castle for his private accommodation. The gardens were re-designed in a C19 interpretation of Tudor style, and this forms the basis of the garden's current appearance. The C19 garden appears to have been laid out on an accumulation of top soil and other material circa 0.8-1m above the earlier garden level. The castle is now (2013) a hotel and restaurant. Trial excavations in 1992 led to the conclusion that there is a strong possibility the early-C16 garden survives largely intact.
In 1679 part of the park immediately to the north of the castle was sold to form a separate estate known by 1775 as Thornbury Park; the present neoclassical house (1832-6, listed Grade II) is now occupied by Sheiling School, Camphill Community, with additional late-C20 residential units to the east of the castle. To the west of the west curtain wall, and outside the registered site, is an area known as the Pithay, now an area of rough grazing. Its historic development is uncertain although 'the Pitties' is referred to in an inventory of 1521 by the Crown Commissioners. Although backfilled with large amounts of building rubble in the 1970s, it was evidently the site of extensive quarrying and excavation, and was possibly the site of part of an abortive scheme by the Duke of Buckingham to cut a canal from the castle to the River Severn to the west. To the south of the castle is the church of St Mary, listed at Grade I. The origins of the church lie in the C12 or earlier, but the building is predominantly C15, with later additions and restoration. The fabric incorporates examples of the Stafford knot, as found in the castle building, and it has been suggested that the tower's openwork crown belongs to the first part of the C15, but there is currently no evidence that the church's surviving fabric contains work directly associated with the 3rd Duke of Buckingham.