In the Late Middle Ages, at least two towered palaces stood at the location, which was delineated around 1500 by the walls of the anti-Turkish fort in Vipava, most of which is still preserved. Their builders, and early owners, probably belonged to Aquileian ministeriales, as mentioned in the sources. The palaces were probably built in the 13th century, and are around the mid-14th century, about the same time as the upper castle, passed into ownership of the Habsburgs. A source from 1401 reports that William, Duke of Austria at that time granted the palace in Vipava a feud to Viljem Baumkircher. It was then that the towered palace, together with the associated estate, got the name Baumkircher Tower.
The Baumkircher Tower became famous because of Andrej Baumkircher, who rose from humble background to one of the most important personalities in the political history of the Austrian lands in the 15th century. For his rise, he skilfully took advantage of disputes between Emperor Frederick III, King of Aquileia Matthias Corvinus and the Counts of Celje. As a skilful soldier, he soon became an indispensable commander of the mercenary troops of Emperor Frederick III (1415-1493). He paid Baumkircher with possessions, privileges and money.
Vipava faces Turkish raids in the years 1476-1478. During the planned fortification of towns and squares in Carniola against the Turks, Emperor Frederick III in 1478 ordered the residents of Vipava to surround their square with walls and moats. Shortly afterwards, the former Baumkircher towers were incorporated into the complex of the new anti-Turkish fort, which was used as a refuge for the local population. This is when the complex got the name Tabor Fort.
