The main part of the castle consists of the family portions. At up to eight stories, these eight towers reach heights of between 30 and 40 metres (98 and 131 ft). They are fortified with strong exterior walls; to the yard, they present a partial framework. About 100 members of the owners' families lived in the over 100 rooms of the castle. A village once existed below the castle, on its south side, which housed servants, craftsmen, and their families supporting the castle. The castle was founded in a region that was an important trade route between rich farmlands and their markets in the Roman period. With the collapse of the Western Roman Empire in the late 5th Century AD, the area was conquered by the Franks. However with the division of Charlemagne's empire the land was given to his son Louis the Pious (814). It was during this period that the site was occupied by a simple manor hall with an earthen palisade.

Several hundred years later, House of Eltz began work on the Platteltz, a Romanesque keep, which was built on the site of the old manor hall. This remains the oldest part of the castle. By 1157 the fortress was an important part of the Holy Roman Empire under Frederick Barbarossa. It continued to protect the trade routes from the Moselle Valley and the Eifel region. Between 1331–1336, the castle endured the only serious military conflicts in its history during the Eltz Feud when lords in the region including Johann of Eltz, together with free imperial knights, opposed the territorial policies of Balduin von Trier who was the Elector and Prince-Archbishop of Trier. As a result of the standoff, Balduin lay siege to Johann at Eltz Castle. It was bombarded by primitive cannons known as Pot-de-fer and catapults from a small siege castle, Trutzeltz Castle, which was built on a rocky outcrop on the hillside above the castle (the remains can still be seen today as a few ruined walls on the northern side of the castle). The siege of Eltz ended after two years when the free imperial knights agreed to accept the laws and sovereignty of Trier. Archbishop Balduin reinstated Johann to the Burgrave, but only as his vassal and no longer as a free knight.

Started in 1470 by Philipp zu Eltz, the 10-story Greater Rodendorf House takes its name from the family's land holding in Lorraine. The oldest part is the flag hall with its late Gothic vaulted ceiling, which was probably originally a chapel. Construction was completed around 1520. The (so-called) Little Rodendorf house was finished in 1540, also in Late Gothic style. It contains the vaulted "banner-room". In 1472 the House of Rübenach completed the castle wing that has Late Gothic architecture. The buildings include the Rübenach Lower Hall, living room, and the bedchamber with its opulently decorated walls.

In 1615 the House of Kempenich house replaced castle's original hall. Every room in this part of the castle could be heated; in contrast, other castles might only have one or two heated rooms. During the Palatinate War of Succession from 1688 to 1689, many of the early Rhenish castles were destroyed. However, the castle was saved from destruction because its lord as the time was Hans Anton zu Eltz-Üttingen who was also a senior officer in the Royal French Army of Louis XIV. Eltz-Üttingen was able to use his position to delete the castle bearing his name from the list of buildings and fortifications to be destroyed.

Count Hugo Philipp zu Eltz was thought to have fled during the French rule on the Rhine from 1794 to 1815. The French confiscated his possessions on the Rhine and nearby Trier which included Eltz castle, as well as the associated goods which were held at the French headquarters in Koblenz. In 1797, when Count Hugo Philipp later turned out to have remained hidden in Mainz, he came back to reclaim his lands, goods and wealth. In 1815 he became the sole owner of the castle through the purchase of the Rübenacher house and the landed property of the barons of Eltz-Rübenach.

In the mid 19th century, Count Karl zu Eltz restored the castle. Between 1845 and 1888, 184,000 marks (equivalent to €2,680,000 in 2021) was spent on extensive construction work that carefully preserved the existing architecture. Extensive security and restoration work took place between the years 2009 to 2012. Among other things, the vault of flags hall was secured after it was at risk of partially collapsing walls and the porch of the Kempenich section. In addition to these static repairs, almost all the slate roofs were replaced. Structural problems were remedied in the ceiling and wood damage was repaired. In the interior, heating and sanitary facilities, windows and fire alarm system were renewed, and also historic plaster was restored. The half-timbered facades and a spiral staircase were renovated at the costs of around €4.4 million. The measures were supported by a €2 million grant from an economic stimulus package provided by the German federal government. The state of Rhineland-Palatinate, the German Foundation for Monument Protection and the owners provided further funds. The Rübenach and Rodendorf family wings in the castle are open to the public, while the Kempenich branch continue to use their third of the castle. Public visits are seasonal: from April to November. Exhibits include the treasury, which contains gold, silver and porcelain artifacts, and the armory which hosts historic weapons and suits of armor.