Diego Velázquez's Portrait of Innocent X (previously Cardinal Giovan Battista Pamphilj, who became Pope in 1644) is considered to be the collection's masterpiece. Velázquez, while not idealizing the Pope's countenance, is not unflattering in the portrait; Innocent X's features were believed by his contemporaries to symbolise a despotic lifestyle and vindictive character. The portrait painted to commemorate the Holy Year was commissioned by his hedonistic sister-in-law Olimpia Maidalchini, who was his close confidante and adviser, and some say mistress. Since 1927, Velázquez's portrait has been placed in a specially designated small room along with a sculptured bust of the same pope by Gian Lorenzo Bernini.

Olimpia Maidalchini's son Camillo Pamphilj, defying his powerful mother, renounced the cardinalship conferred on him by his uncle the Pope and married the widowed Olimpia Borghese. Born an Aldobrandini, she brought the palazzo known as Palazzo Aldobrandini into the Pamphilj family. Following a period of exile in the country, to avoid confrontation with the Pope and Olimpia Maidalchini, the newly married couple took up permanent residence in the Palazzo Aldobrandini, which from 1654 Camillo began to expand on a large scale; neighbouring houses and a convent were bought and demolished as the palazzo grew, in spite of local opposition from the neighbouring Jesuits at the Collegio Romano. The architect in charge of this lengthy project was Antonio Del Grande. The façade facing the Via del Corso, however, is by Gabriele Valvassori. Following Camillo's death in 1666, building continued under the auspices of his two sons Giovanni Battista (his heir) and Benedetto.

One of Camillo and Olimpia's daughters, Anna Pamphilj, married the Genoese aristocrat Giovanni Andrea III Doria Landi in 1671, and it was their descendants who inherited the Palazzo when the Roman branch of the Pamphilj family ended in 1760. In 1763 Prince Andrea IV combined his Genoese and Roman names to the present Doria-Pamphilj-Landi. In 1767 the ceilings of the state rooms were frescoed by late-Baroque artists such as Crescenzio Onofri, Aureliano Milani, and Stefano Pozzi (Sala degli Specchi).

The collection was first opened to the public by the three-quarters English Princess Orietta Pogson Doria Pamphilj, whose English husband Commander Frank Pogson added her name to his. Her own father, Prince Filippo Andrea VI, was half-English. Princess Orietta and Commander Frank did much to restore the collection and the palazzo; following her death in 2000 the guardianship of the collection was taken over by her adopted, English-born children, Jonathan Doria Pamphilj and Gesine Pogson Doria Pamphilj, who still live in the palazzo. Along with the possessions of the Colonna and Pallavicini-Rospigliosi families, this is one of the largest private art collections in Rome.