Red Bay is a natural harbour residing in the bay that gives it its name, both names in reference to the red granite cliffs of the region. Because of the sheltered harbour it was used during World War II as a mooring site for naval vessels. In the bay are Penney Island and Saddle Island, which were used by the Basques for their whaling operations. The location of the sunken vessel San Juan is near Saddle Island.
Between 1550 and the early 17th century, Red Bay, known as Balea Baya (Whale Bay), was a centre for Basque whaling operations. Sailors from southern France and northern Spain sent 15 whaleships and 600 men a season to the remote outpost on the Strait of Belle Isle to try to catch the right whale and bowhead whales that populated the waters there, according to Memorial University of Newfoundland.
In 1565, a ship—believed to be San Juan—sank in the waters off Red Bay during a storm. Other, smaller vessels, such as chalupas, have also been recovered from the waters.
Another galleon was found 25–35 feet below water in 2004. It was the fourth trans-oceanic ship to have been found in the area.
A cemetery on nearby Saddle Island holds the remains of 140 whalers. Many of the people buried there are thought to have died from drowning and exposure.
Historians believe that a decline in whale stocks eventually led to the abandonment of the whaling stations in Red Bay. Today, an interpretive centre in Red Bay explains the history to visitors.
Local legends of Red Bay make reference to a hidden treasure buried in a body of water known as Pond on the Hill 51°43′43″N 56°26′56″W / 51.72861°N 56.44889°W / 51.72861; -56.44889 at the foot of Tracey Hill by the infamous pirate Captain William Kidd. An attempt was made to find the treasure by residents of Carrol Cove by draining the pond. The attempt had failed.
Red Bay has been designated a National Historic Site of Canada since 1979, and since 2013 it is one of Canada's UNESCO World Heritage Sites.