Prem Castle was first mentioned in written sources in 1213, with Udo de Prem, a ministerial from Devin (Duino) being mentioned as a witness. After the male line of the Lords of Devin died out in 1399, Prem and its extensive estate was inherited by their relations, the Lords of Walsee from Upper Swabia. After the Counts of Celje died out in 1456, the Habsburgs increased their power and the Lords of Walsee were forced in the period between 1466 and 1472 to sell their strategically most important estates of Devin, Štivan (San Giovanni di Duino), Senožeče and Prem to Emperor Frederick III. At the beginning of the 17th century the renovated castle passed through marriage into the hands of an influential noble family, which moved to Carniola from Pordenone in Friuli in the second half of the 16th century as part of the Counter-Reformation. Prem remained the property of the Porcia family until 1906, when the estate was auctioned off. It was bought by the curate and later parish priest in Dolnja Košana, Karel Lenassi. After his death Prem Castle and the manor in Senožeče were inherited by his niece Ana Biščak. During World War One, the manor and many other buildings behind the front line were occupied by Austro-Hungarian soldiers who ransacked its interior. It is a miracle that both the castle and the manor in Senožeče were not demolished after the war.

In 1927 the decaying castle was bought by a doctor from Trieste called Bruno Zuccolin, who spent several years thoroughly renovating it according to plans drawn up by the painter Mario Lannes, and turned it into his summer residence. He remained its owner until it was nationalised after World War Two. He had various stone fragments of coats of arms and bosses of Venetian origin built into the wall around the entrance to the ground floor room under the chapel, but these have nothing to do with the castle’s past. It was probably also in the same period that two brackets in the form of Late Renaissance herms, also from the region of the former Republic of Venice, were fitted to the wall of the first floor loggia.

After the war, the castle – like all such buildings – was nationalised. Major renovation work was carried out from 1970 to 2008 as it had again decayed considerably in the post-war period.