According to a legend, a local church (Chiesa dei Novelli Innocenti) was founded in the early 14th century in honour of hundreds of participants to the so-called Children's Crusade of 1212 who perished in a shipwreck just off the island on their way to North Africa during a gale. The church, called Chiesa dei Novelli Innocenti and is located within the town perimeter, is not currently used as a church (only one time in the year); it was the only evident remainder of building found at the time of colonization in 1739.
Carloforte was founded in the 18th century by around 30 families of coral fishers, originally from the Ligurian town of Pegli, near Genoa. They had left their hometown in 1541, and had settled in the island of Tabarka, off the coast of Tunisia, to fish for coral. After centuries, the coral in that area was exhausted and the families, while setting off back to Italy, found there was plenty of coral in the sea off the Sardinian west coast. They asked the King of Piedmont-Sardinia Charles Emmanuel III for permission to settle down on the once uninhabited San Pietro Island instead. When he granted them permission, the island was colonized (1739); the name Carloforte ("Charles the Strong", but also the "Carlo's Fort") was given to the town they then proceeded to found, in the Piedmontese king's honour. To this day, Carloforte maintains strong cultural ties with the mainland towns of Pegli and Genoa: the population still speaks a variety of Ligurian language called tabarchìn (or tabarchino, in Italian), separate from both Italian and Sardinian, which is used even by most children and taught in the island's schools.
Modern-day Carloforte's principal sources of revenue are fishing, tourism, and remittances from the many merchant mariners around the world who hail from Carloforte.
Carloforte, boasting several beaches, both rocky and sandy, is a summer tourist destination. On the other side of the island from Carloforte is a functioning lighthouse (Capo Sandalo Lighthouse, now automated), locally known simply as Il Faro and whose structure dates back to the 1880s.
Carloforte boasts a long-standing tradition of wooden boatbuilding. Locally built traditional lateen sail fishing boats (luggers) can be spotted in the harbour.
To the north of the town (at the La Punta locale) is a collection of buildings that housed the former communally owned tuna-processing plant linked to the once-vibrant tuna fishing tonnare industry on the island. Some of the buildings have undergone renovation to some extent and now contain wind-surfing school and a new, privately owned tuna processing business. Some of the buildings are still in ruins, but remain of great architectural interest.
The town hosts a maritime high school.
Ferry services connect the port of Carloforte with the Sardinian mainland at Portovesme as well as Calasetta, a similar fishing port on Sant'Antioco Island.