At some time after 1390 the medieval core of the present structure was built by Sir Philip Courtenay (d. 1406), the 5th or 6th son of Hugh Courtenay, 2nd Earl of Devon (d. 1377). The Earls of Devon were seated at Tiverton Castle until 1556, and their cousins of this cadet line known as "Courtenay of Powderham" continued to exist in parallel, not always on amicable terms, as prominent county gentry, arguably the leading and most prestigious gentry family of Devon, actively engaged in the local administration of Devon as JP's, sheriffs and MP's. From 1556, with the extinction of the senior line of Courtenay of Tiverton, the Courtenays of Powderham became de jure Earls of Devon, and became de facto Earls from 1831 when the title was confirmed to them in law. They had, however, obtained the right to sit in the House of Lords when created Viscounts in 1762.

The original building on the site was a fortified manor house, the appellation "castle" was added probably no earlier than the 17th century. The building has never been a true castle, that is to say with a keep and moat, although it did possess a curtain wall and yard on the east side (now the rose garden) as shown in the 1745 engraving by Buck. Leland mentioned a barbican or bulwark in this area, but these were demolished as part of the 18th-century landscaping works designed to provide an uninterrupted view from the lower rooms towards the Exe Estuary. Many castle-like elements on the west front (main entrance) were added in the 19th century. The gatehouse was built between 1845 and 1847 to a design by Charles Fowler. The tall rectangular structure beyond with a tower to the north is essentially the original fortified manor house. The projection from the lower storey to the north in lighter stone with three Gothic-style windows is the Victorian dining hall, designed by Fowler.

The house is centred on the 14th- and 15th-century thickly-walled double-height rectangular building formerly comprising from north to south the withdrawing room, great hall, screens passage and kitchens, which are now represented in the same orientation by the ante-room, Staircase hall, Marble hall and Victorian kitchen.

The Marble Hall, named from its black and white marble floor, was completed in 1755 and forms the lower and southern part of the former medieval great hall, which was divided by an internal wall in the early 18th century into Staircase hall and Marble hall. Originally it was double height, as high as the staircase hall to the north, before the ceiling was added in the 18th century to form bedrooms above. At the same time the staircase was inserted into the upper part to form the staircase hall.

The screens passage was located in this end of the hall. The timber screen which formed the north side of the screens passage was demolished at the time of the partition, but three medieval Gothic-arched doorways through the south stone wall of the screens passage into the kitchen remain. A single more flatly arched doorway remains high up on the south wall, which formed the entrance to the wooden minstrels' gallery overhanging the great hall.

The Marble Hall is used as a sitting room which has an 18th-century fireplace. Contents of the room include a 14 feet (4.3 m) high longcase clock made about 1745 by William Stumbels of Totnes; a large 17th-century Brussels tapestry with rustic farm-yard scenery after Teniers above the fireplace; and a 1553 carved wooden over-mantel decorated with the Courtenay arms. Two portraits of the present Earl of Devon and his wife hang on the north wall above the wooden panelling.

The house has a mixture of medieval features and fine 18th-century decoration. William Courtenay, 11th Earl of Devon (d. 1888) installed a heraldic chimneypiece in the Dining Hall in memory of his grandfather Reginald Courtenay (1741–1803), Bishop of Exeter from 1797 to 1803, and of his parents. The Dining Hall was built by his father William Courtenay, 10th Earl of Devon (d. 1859) between 1847 and his death in 1859, and the 11th Earl completed the internal decorations in 1860, including the linen fold panelling containing several dozen ancestral heraldic shields.[10] It is copied from the medieval chimneypiece in the Bishop's Palace, Exeter, installed c. 1485 by Peter Courtenay (d. 1492) Bishop of Exeter, a younger son of Sir Philip Courtenay (1404–1463) of Powderham.