The Palacio de Villena was an aristocratic residence built for Antonio Velasco y Rojas in the mid-16th century and is opposite the College of San Gregorio.
After being inherited by various generations, ownership of the building fell to the wife of the Marqués de Villena in the 18th century (and the reason the house is known by that name) and later to the Duques del Infantado, the Marqués de Casa Pombo, and the Marqueses de Alonso Pesquera, who eventually sold it to the State in 1919. From then on, and until 1982, the palace housed the seat of the Civil Government.