The first European settlers on the island came from Duchy of Courland and Semigallia, a vassal state of the Polish–Lithuanian Commonwealth, who also had other colonial possessions in the area, though the English Crown had granted the island to two separate companies in 1588 and 1618. In 1651, the settlers built a fort that they named Jacob Fort after Jacob Kettler, the Duke of Courland, and used it as a trade base. The Dutch briefly held the fort from 1659 until the English captured it in 1661; the Dutch ceded the fort to the English in 1664.
The English renamed the island James Island and the fort Fort James after James, the Duke of York, later King James II of England. The chartered Royal Adventurers in Africa Company administered the territory, which used it first for the gold and ivory trade, and later in the slave trade. On 1 August 1669, the company sublet the administration to the Gambia Adventurers. In 1684, the Royal African Company took over the Gambia's administration.
In 1695, the French captured Fort James after a battle with English sailors. In 1702, Fort James was definitively under British control. The fort was destroyed and rebuilt several times in this period, both in conflicts between the English and French and by pirates. On 13 June 1750 the Company of Merchants Trading to Africa assumed the administration of the Gambia. Between 1758–1779, the Gambia was part of British Senegambia.
The Six-Gun Battery (1816) and Fort Bullen (1826), now included in the James Island UNESCO World Heritage Site and located on both sides of the mouth of the River Gambia, were built with the specific intent of thwarting the slave trade once it had become illegal in the British Empire after the passing of the Slave Trade Act in 1807. These sites along with the island itself were abandoned in 1870.
On 6 February 2011, at the request of New York Artist Chaz Guest to Gambian President Yahya Jammeh, it was renamed to Kunta Kinteh Island to give the island a Gambian name. At the renaming ceremony, Guest revealed the miniature replica of his 9-m (30-foot) statue of Kunta Kinte that would be displayed on Kunta Kinte Island.
As an important historical site in the West African slave trade, it is listed as a UNESCO World Heritage Site, together with related sites including Albreda, Juffureh and Fort Bullen. Kunta Kinteh Island is suffering heavy erosion, and is now approximately 1/6 of the size during the time when the fort was active. Ruins of several of the British administrative buildings (including a single cell, apparently used to house the most troublesome captives), a small jetty and a number of skeletal baobab trees remain. The ruins have been stabilised and protected by a capping. Because the island is low-lying, during high tide and storms sometimes waves will beat against some of the surviving structures.
Kunta Kinte, a character described in Alex Haley's book and TV series Roots, has become associated with James Island. The book states that Kunta Kinte was among 98 slaves that the slave ship Lord Ligonier brought to Annapolis, Maryland in 1767.