The Crusaders, who founded the Latin Kingdom of Jerusalem in 1099, did not at first succeed in overcoming Akko’s fortifications. On 26 May 1104, after months of heavy siege and with the help of the Genoese fleet, the city surrendered and was handed over to King Baldwin I. Aware of the significance of the city and its port for the security of their kingdom, the Crusaders immediately began to construct a sophisticated system of fortifications composed of walls and towers, unlike any built previously. These fortifications were built along the sea to the west and south of the city, while in the east and north a mighty wall (probably a double wall) with a broad, deep moat separated the city from the mainland. The port was also rebuilt and, according to literary sources and maps, included an outer and an inner harbor (the latter now silted). A new breakwater was built, protected by a tower at its far end; it is today known as the Tower of Flies.

The fortifications of Akko, in which the Crusaders had placed their trust, fell relatively easily to the Muslims. Shortly after their victory at the Battle of the Horns of Hattin, on 9 July 1187, the city surrendered to Salah al-Din (Saladin) and its Christian inhabitants were evacuated.