The castle is located on a point dominating the Dordogne River. Excavations found pottery ovens, indicating that the site had a large population in Gallo-Roman times, and that there was a wooden fort on the site, and later a Gallo-Roman oppidum, or fortress.
The castle was built in the 11th century to protect the city of Bordeux. It was made of stone and surrounded by a wooden palisade, as attested in a manuscript dated 1092. Nothing remains of the original structure.
In 1288 the chateau and lands were given as a dowry to Amanieu VII d'Albret, who became the seigneur des Vayres. He strengthened the chateau with a new tower, Tour du Moulin. The chateau remained in his family for three hundred years, until the time of King Henry IV.
Beginning in 1326 Bérard d’Albret took the side of England in the struggle for control of the region. He received financial support from King Eward II of England, and turned the castle into a large fortress. From that period, the donjon, the entry and the moats, now dry, still remain.
The château was badly damaged during the Hundred Years War. During the 14th century, its successive owners switched back and forth between the French and English side. The castle was seized in retaliation and given to different nobles, including Gaston de Foix, and another branch of the d'Albret family. In 1499 it was given by marriage to Cesar Borgia, whose daughter then restored it to Henri d'Albret, the king of Navarre and the grandfather of the future King of France, Henry IV, in 1535.
Henry IV inherited the house through his mother, Jeanne d'Albret. He stayed in the castle several times, and then in 1583 he sold it, largely ruined, to Ogier de Gourgue. Ogier de Gourgue was in charge of the treasury of Guyenne, and he commissioned a well-known architect, Louis de Foix, who had built the lighhouse of Cordouan and had worked for the king of Spain, to rebuild the castle in the new style of the French Renaissance, with an elegant facade, in the mannerist style of the late Renaissance, facing the Court of Honor.
In the 17th century the Château de Vayres was the scene of battles between the Gourgue family and other nobles the region against the power of Cardinal Mazarin and the young King Louis XIV. The chateau was badly damaged during these battles.
In about 1700 Jacques-Joseph de Gourgue, the bishop of Bazas, undertook the restoration of the chateau in the style of the eighteenth century. He harmonized the structures and created a monumental stairway on the site of the old moats. In the beginning of the 17th century he replaced the drawbridge and the barbacane with the present bridge and portico in the style of Vauban, by which one today enters the chateau. The chateau has not been restored since that time.
The Gourgue family were the owners of the chateau until 1900. In 2001, the chateau was classified as an historical monument of France. The castle itself is still occupied, and has undergone considerable restoration inside.
The original gardens were created in the 17th century, were large and extended along the Dordogne River. They were abandoned and replaced by fields, and then recreated in 1938 as a Garden à la française by the landscape architect Louis-Ferdinand Duprat. A monumental stairway leads from the chateau across the old moat to the French gardens by the river, where there are parterres bordered with hedges of yew, and boxwood trees clipped into cone shapes. There is also a flower garden of medieval inspiration, and an English-style park, with cedar, oak, linden, hornbeam and copper beech trees.