A house of note is Stafford House which stands a short distance from the village. This was the seat of the Floyer family and one of the manors within the parish. It was the principal manor at Frome Billet alias Frome Everard, known as Everard's Manor, through which family it passed for several generations until after the dissolution when it was granted to Edward Neville. The advowson of the church went with this manor and both passed into the hands of the Long family in 1613. The manor was subsequently conveyed to John Gould, a merchant of Dorchester. John Gould built a new house of Portland Stone in 1633 that incorporated a wing of an earlier one and it remained in this family until 1831. At this time it was sold to John Floyer whose name appears on the later census residing at Stafford House.

The Manor of Stafford also known as Bingham's Manor, lay to east of the parish. It was held mainly by the Bingham family of Bingham's Melcombe until 1571 when Robert Bingham sold it. At that time it consisted of a farm and six tenements. In 1638 it was in the possession of the heirs of Richard Russel, the deceased rector of Stafford. Passing down the line of Russells and his great-grandson Richard dying in 1671 at the age of four years, it passed to the young Richard's aunt, Anne Russell who had married in 1650 to Robert White of Winterborne St Martin (Martinstown). Their son, Richard inherited and became High Sheriff of Dorset in 1714. George, the son of Richard, inherited but died without issue in 1749 at which time it went to George Acton, the son of the Rev. Edward Acton who had married Susannah White, sister of George. In 1780 it was sold by the Acton family to John Floyer of Upwey. The house itself was substantially altered around 1720, but remains today as the Manor House at Manor Farm.

The Floyer Family

At Stinsford in 1741, John Floyer (d.1756) of Upwey married Anne (d.1746), the daughter of James Richards and sister and heiress of George Richards who was Lord of the Manor of West Knighton. Their younger son, William (1746-1819) became Rector of West Stafford and Vicar of Stinsford, while his elder brother John (1744-1789), married in 1776 to Jane (d.1808), the widow of a former rector, James Acton who had died in 1775. This John purchased his interest in the manor that was to descend down through the family for some time to come. John died without issue and the estate passed to his nephew, John Floyer the son of his brother William, who purchased Stafford House in 1831.

The parish church of West Stafford is dedicated to St. Andrew. In part dating from the fifteenth century and rebuilt around 1640. A chalice dates from 1638 and a fine chandelier from 1713.

Of the monuments inside the church, near the altar is one to a former rector, Canon Reginald Southwell Smith, who 'reigned' almost as long here as Queen Victoria was on the throne and at the same time. His three daughters, Emily, Alice and Evangeline, kept diaries of life in the parish from 1836 to 1933 and they are to be found in the Dorset History Centre.

Another monument, bearing the arms of Russell, Dukes of Bedford, commemorates four Richard Russells, three of whom were rectors of the parish when they died in 1638, 1660 and 1667 and also the son of the last who died in 1671 aged 4 years.

The old rectory stands a short distance from the church with an old tithe barn, both once thatched, now known as Glebe Court. It dates from around the rebuilding of the church and was rebuilt in 1767. It is said to contain an old desk which the poet, Wordsworth, used to write his poetry.

The cottages and houses that line the roadside through the village date mainly from the 18th and 19th centuries built of cob and thatch.

In the early hours of 30th April 2006, fire broke out at the historic 400 year old Wise Man Inn. Firefighters used water from the nearby river to douse the flames and although the building was a ruin, the owners of the pub promised it would be rebuilt and it has. The picture below was taken before the fire.

The inn dates back to the seventeenth century and was formerly cottages, built of stone and thatch. The cottages were converted to a pub in 1937. Where the name of it comes from is something of a mystery, but a poem on a wall is attributed to Thomas Hardy, though others attribute it to Goldring in 1850. A John Goldring is listed in trade directories of 1855, 1859 and 1875 as a beer retailer as well as a grocer. Perhaps this was the beginning of the establishment of the Wise Man Inn. Later directories give a Miss Rosalie Goldring as a beer retailer and she was also an overseer of the parish and a collector of taxes.