The Castle Hill has been settled from the 5th millennium BC on. Settlers of Roman Poetovio built there a fortress and temples; in the early Middle Ages the Hill became the burying ground of the Slavs.

The medieval fortress was built in the 11th century, the period when the Castle and the town were in the hands of the Salzburg archdiocese. Archbishops let the Castle out to the Lords of Ptuj, who founded, during their three-hundred-year-long stay in Ptuj, the Dominican and Minorite monasteries, and built the pilgrimage church at Ptujska Gora. The gravestone of the last Lord of Ptuj, Friedrich IX, is built in the wall of the Castle's ground floor. Between 1656 and 1802, the Castle was home to the Leslie family. Its last owners, counts Herberstein, lived in the Castle from 1873 to 1945, when the Castle was nationalised and became, together with its rich furniture, a component part of the Ptuj Museum.

Designed in two parts, the medieval fortress on the top of the Hill disposed of residential and outdoor buildings. The western flat part of the crest was built with military buildings, among which the Western Tower, the oldest preserved part of the Castle complex. In the 12th century, the central crest of the hill was occupied by a larger group of buildings including the palatial one, as well as the mighty defence tower called Konrad's tower. In the 14th century, the central castle building got the second floor and was designed in a form of a horse-shoe; the castle remparts were

connected with those protecting the town. The appearance of the Castle was dramatically changed in the 16th century by Italian master-builders, who were commissioned to fortify the town and the Castle to withstand Turkish invasions. In the 17th century, the Turkish threat diminished which resulted in the demolition of the Konrad's tower. The castle was enlarged, horse stables and a tower were constructed on the eastern part of the hill top. The 18th century witnessed the construction of an administrative building along the southern wall, of the castle granary on the western platform, and the so-called »tournament field« below it.