The Nelson House in Yorktown was built by the merchant "Scotch Tom" Nelson, who, despite his wealth, was never accepted as the social equal of the members of the Virginia planter aristocracy.