The nucleus of the estate was a small farm of 100 acres (0.40 km2), called Cartleyhole, nicknamed Clarty (i.e., muddy) Hole, and was bought by Scott on the lapse of his lease (1811) of the neighbouring house of Ashestiel. He first built a small villa and named it Abbotsford, creating the name from a ford nearby where previously abbots of Melrose Abbey used to cross the river. Scott then built additions to the house and made it into a mansion, building into the walls many sculptured stones from ruined castles and abbeys of Scotland. In it he gathered a large library, a collection of ancient furniture, arms and armour, and other relics and curiosities, especially connected with Scottish history, notably the Celtic Torrs Pony-cap and Horns and the Woodwrae Stone, all now in the Museum of Scotland.
The last and principal acquisition was that of Toftfield (afterwards named Huntlyburn), purchased in 1817. The new house was then begun and completed in 1824.
The general ground-plan is a parallelogram, with irregular outlines, one side overlooking the Tweed; and the style is mainly the Scottish Baronial. Into various parts of the fabric were built relics and curiosities from historical structures, such as the doorway of the old Tolbooth in Edinburgh.
Scott had only enjoyed his residence one year when (1825) he met with that reverse of fortune which involved the estate in debt. In 1830, the library and museum were presented to him as a free gift by the creditors. The property was wholly disencumbered in 1847 by Robert Cadell, the publisher, who cancelled the bond upon it in exchange for the family's share in the copyright of Sir Walter's works.
Scott's only son Walter did not live to enjoy the property, having died on his way from India in 1847. Among subsequent possessors were Scott's son-in-law, John Gibson Lockhart, J. R. Hope Scott, QC, and his daughter (Scott's great-granddaughter), the Hon. Mrs Maxwell Scott.
The house was opened to the public in 1833, but continued to be occupied by Scott's descendants until 2004. The last of his direct descendants to hold the Lairdship of Abbotsford was his great-great-great-granddaughter Dame Jean Maxwell-Scott (8 June 1923 - 5 May 2004). She inherited it from her elder sister Patricia Maxwell-Scott in 1998. The sisters turned the house into one of Scotland's premier tourist attractions, after they had to rely on paying visitors to afford the upkeep of the house. It had electricity installed only in 1962. Dame Jean was at one time a lady-in-waiting to Princess Alice, Duchess of Gloucester, patron of the Dandie Dinmont Club, a breed of dog named after one of Sir Walter Scott's characters; and a horse trainer, one of whose horses, Sir Wattie, ridden by Ian Stark, won two silver medals at the 1988 Summer Olympics.
In 2005, Scottish Borders Council was considering an application by a property developer to build a housing estate on the opposite bank of the River Tweed from Abbotsford, to which Historic Scotland and the National Trust for Scotland objected.
Sir Walter Scott rescued the "jougs" from Threave Castle in Dumfries and Galloway and attached them to the castellated gateway he built at Abbotsford.
Tweedbank railway station is located near to Abbotsford House.
Abbotsford gave its name to the Abbotsford Club, founded by William Barclay Turnbull in 1833 or 1834 in Scott's honour, and a successor to the Bannatyne and Maitland Clubs. It was a text publication society, which existed to print and publish historical works connected with Scott's writings. Its publications extended from 1835 to 1864.
In 2012, a new Visitor Centre opened at Abbotsford which houses a small exhibition, gift shop and Ochiltree's Dining, a cafe/restaurant with views over the house and grounds.
In 2014 it won the European Union Prize for Cultural Heritage / Europa Nostra Award for its recent conservation project.