The settlement at Ko Panyi was established at the end of the 18th century by nomadic Malay fisherman. Ko Panyi is known as Pulau Panji in Malay language. It was during this time that the law limited land ownership solely to people of Thai national origins, and due to this fact the settlement was, for the most part, built on stilts within the protection of the island's bay, providing easy access for fishers. With the increase of wealth for the community, due to the growing tourism industry within Thailand, purchase of land on the island itself became a possibility, and the first structures of relevance were built, a mosque and a freshwater well.
The village has a Muslim school which is attended by both girls and boys in the mornings. Due to the informal nature of this education, many of the boys attend schools further afield in Phang Nga or in Phuket. Further emigration from the village is encouraged as the size of the settlement is restricted by dangerous water conditions in the rainy season.
A mosque based on the island adjacent to the settlement serves the predominantly Muslim population and is a focal point and meeting place for the community. A market stocked with goods from the mainland sells basic amenities such as medicine, clothes and toiletries.
Despite the recent rise in tourism, life in Ko Panyi is still primarily based around the fishing industry as tourists only visit in significant numbers during the dry season.
The village includes a floating football pitch. Inspired by the 1986 FIFA World Cup, children built the pitch from old scraps of wood and fishing rafts. After making it to the semi-finals in an inland tournament, all the village were inspired to take up the sport. They built a brand new pitch, although the wooden one still remains and is popular among tourists. As of 2011, Panyee FC is one of the most successful youth soccer clubs in Southern Thailand, and the boys who built the pitch back in 1986 are now grown men. A 2011 brand campaign for TMB Bank includes a short film that tells the team's story. The film is based on interviews with the original team, and it stars local children rebuilding the field on location.
In the late-20th century, the community found it difficult to subsist solely on the fishing industry and the postman proposed to invite tourists to the village to benefit the residents. Nowadays this is one of the main attractions on tours of Phang Nga Bay from Phuket, often serving as a lunch stop. With the increasing number of tourists, a number of seafood restaurants are present on the island, as well as various stalls selling souvenirs. In addition, their legendary football team's old pitch serves as a major attraction. The village became a pit stop during the fourth leg of the 19th season on the U.S. reality-competition show The Amazing Race.