According to the White House Historical Society and Dolley's personal letters, President James Madison left the White House on August 22 to meet with his generals on the battlefield, as British troops threatened to enter the capitol. Before leaving, he asked his wife Dolley if she had the "courage or firmness" to wait for his intended return the next day. He asked her to gather important state papers and be prepared to abandon the White House at any moment. The next day, Dolley and a few servants scanned the horizon with spyglasses waiting for either Madison or the British army to show up. As British troops gathered in the distance, Dolley decided to abandon the couple's personal belongings and save the full-length portrait of former president and national icon George Washington from desecration by vengeful British soldiers, many of whom would have rejoiced in humiliating England's former colonists.
Dolley wrote to her sister on the night of August 23 that a friend who came to help her escape was exasperated at her insistence on saving the portrait. Since the painting was screwed to the wall she ordered the frame to be broken and the canvas pulled out and rolled up. Two unidentified "gentlemen from New York" hustled it away for safe-keeping. (Unbeknownst to Dolley, the portrait was actually a copy of Gilbert Stuart's original). The task complete, Dolley wrote "and now, dear sister, I must leave this house, or the retreating army will make me a prisoner in it by filling up the road I am directed to take." Dolley left the White House and found her husband at their predetermined meeting place in the middle of a thunderstorm.
The next night, August 24, British troops enjoyed feasting on White House food using the president's silverware and china before burning the building. Although they were able to return to Washington only three days later when British troops moved on, the Madisons were not again able to take up residence in the White House and lived out the rest of his term in the city's Octagon House. It was not until 1817 that newly elected President James Monroe moved back into the reconstructed building.