On September 11, 1161, Emperor Frederick grants the Aviano castle to the Bishop of Belluno. The fortress therefore must have been erected before that date, perhaps being constructed on a site already fortified during Roman times.
In 1328 the castle was granted to Pietro de Rubeis and then in April 1334 to Morando, Odorico and Nanfosis of Porcia upon payment of a large sum of money. From 1337 the castle, passed to the Savorgnan family, was involved in the wars the Camino family waged against the Patriarchs. On September 11, 1387, during the war which broke out after the nomination of Patriarch Philippe de Alençon, the fortress was conquered by the Carraresi, Lords of Padua. A few years later in 1411, Pippo Spano, King Sigmund’s general captain, occupied the castle with 14 thousand Hungarian cavalrymen. Finally destroyed in 1420 by the Venetian army, it was reconstructed only to be destroyed again during the Turkish incursions.