On a warm spring afternoon in 1861, Mary Anna Custis Lee stood at the entrance of the finest house in Virginia, looked at the furniture, the portraits, her husband’s letters, and her father’s silver, and made a decision that shaped her family’s path for the next sixty years. She took only what she could carry. She believed she would return within weeks. She never set foot in that house again.
Her husband was Robert E. Lee, and the house was Arlington. What happened to the seven children and the wife he left behind did not end with the Confederate surrender at Appomattox. Their story, one could argue, began at that very moment.
One of their daughters would not live to see the war end. One son would spend seventeen years fighting the United States government in courtrooms, not with a rifle this time, but with a lawsuit that reached the Supreme Court. Another daughter disappeared into Europe for decades, and when she returned home, she was arrested. By the end of this story, it becomes clear why historians say the real war of the Lee family did not end in 1865.
It merely changed its shape. Start with the mother, because her story sets the tone for everything that followed. Mary Anna Custis Lee was already crippled by arthritis when Union soldiers crossed the Potomac and occupied her family’s estate. She left Arlington believing the separation was temporary, just as most people in April 1861 believed everything about the coming war was temporary.
Within three years, the federal government turned her front yard into a cemetery for Union war dead, a deliberate punishment aimed directly at her husband, so that no Confederate general could look at his home again without seeing rows of the men his army had killed. For the rest of the war, Mary Lee lived as a refugee, moving between relatives’ homes across Virginia, confined to a wheelchair, watching her husband’s reputation grow into legend while her health quietly deteriorated. When Richmond finally fell and the Confederacy dissolved, she did not get her house back. She got silence and distance, and a husband who spent his last years as a college president in Lexington instead of a plantation owner at Arlington, because Arlington no longer belonged to either of them in any way the law recognized.
Robert E. Lee himself lived only five years after the surrender. He died in 1870 without ever recovering the estate he had spent thirty years restoring. That fact alone would have ended most family stories.
For the Lees, it was only the beginning of the second chapter. There is an irony buried in the grounds of Arlington that most people do not know, and it makes the family’s loss even harder to accept. While the front garden filled with graves, the federal government also set aside part of the estate for a very different purpose: a settlement for formerly enslaved people who had fled bondage during the war. For a time, the hill itself, where Mary Custis Lee had once presided over one of the largest enslaved workforces in Virginia, became home to hundreds of newly freed men, women, and children building their own lives on land the Lees no longer controlled.
History rarely offers such a precise symbol unintentionally, but Arlington did, whether the Lee family wanted to face it or not. Here is the detail almost no one expects. Not all of Lee’s children survived to see the end of the war, let alone its aftermath. His daughter Annie died of typhoid fever in 1862, three years before Appomattox, and was buried in North Carolina in a grave her parents did not see for years.
She never watched her father surrender. She never lived through Reconstruction. In a story about what happened after the fall of the Confederacy, her absence is itself the point: a reminder that loss for this family did not politely wait for the war to officially end before arriving. The two daughters who survived into peacetime carried their own quiet sorrows.
Agnes Lee, the third daughter, had been in poor health for years, and by the early 1870s the struggle was constant. She never married. Neither did her sisters Mary and Mildred, a fact contemporaries noticed and whispered about, as if the choice of four daughters from one of the South’s most prominent families to remain unmarried required special explanation. In October 1873, Agnes died in Lexington at the age of thirty-two.
Her mother, worn down by decades of illness and grief, followed her to the grave just five weeks later. Mother and daughter died in the same autumn. That detail turns the Lee family’s postwar story from a single collapse into something more like a slow, sustained unraveling that kept claiming family members for years after the last shot was fired at Appomattox. But the story does not stay quiet for long, because the eldest daughter refused to let it.
Mary Custis Lee, the family’s firstborn, took a completely different path from her sisters. While Agnes and Mildred stayed close to home and retreated from view, Mary spent most of her adult life traveling across Europe, deliberately distancing herself from the rest of the family, absent from funerals and weddings, living almost entirely on her own terms in an era when that was nearly unheard of for a Southern woman of her standing. For years, to most people, she was like a ghost of the family, present in name but absent in reality. Then, in June 1902, nearly forty years after her family fled Arlington, Mary Custis Lee did something that made headlines across the country.
While riding a streetcar in Alexandria with her Black maid, she sat in the section legally designated for Black passengers under Virginia’s new segregation laws. When the conductor ordered her to move to the white section at the front of the car, she refused. She was arrested. The daughter of the Confederacy’s most legendary general spent the night explaining her position to police, while crowds of Confederate veterans gathered outside the station, unsure whether to treat the moment as a scandal or a spectacle.
To this day, historians still argue about why she did it. Some insist it was a genuine act of quiet defiance against segregation. Others argue she simply did not know the law existed and was too stubborn to move her luggage. Either way, the daughter of the man whom the Lost Cause had turned into a marble saint gave the country a moment no one could fully explain then, and no one has fully explained since.
While Mary made headlines in Alexandria, her younger sister Mildred was living a life that was almost a mirror image of restless travel, moving through Europe, Egypt, and the American West for years without settling or marrying, seemingly determined to build a life as far from the wreckage of Arlington as geography allowed. Family letters describe her as the most tender and cheerful of the four sisters in childhood, the one her father gave a special nickname. Yet she was also the sister who spent the least time as an adult near the family she supposedly adored. She eventually died abroad, in France, in 1905.
Another Lee daughter spent her final decades choosing distance over roots, leaving behind a pattern that by then looked less like coincidence and more like a shared, unspoken family instinct to simply keep moving. Now to the sons, because their struggles with the past bore no resemblance to their sisters’. And those struggles produced the most significant outcome for the entire family. George Washington Custis Lee, the eldest son and his father’s successor in spirit, if not entirely in name, inherited something far heavier than money when Robert E.
Lee died in 1870. He inherited his father’s job. Custis became president of what is now Washington and Lee University, a position he held for more than two decades, walking the same campus grounds where his father had spent his final years trying to rebuild a broken region through education rather than weapons. But Custis also carried something else: a legal claim to Arlington itself, inherited from his mother when she died in 1873.
Unlike his sisters, he chose to fight for it in the only field still open to a defeated Confederate family: the American courtroom. The case dragged on for years, working its way through the federal court system while Custis continued to teach, administer, and wait. It finally reached the United States Supreme Court in 1882, seventeen years after the guns fell silent, and the ruling stunned almost everyone who had assumed the matter was settled the moment Union soldiers buried their first dead on that hill. The Court found that the federal government’s seizure of Arlington had been improper, that the wartime tax sale used to justify the takeover was void, and that legal ownership of Arlington belonged to Custis Lee, and had always belonged to him.
For a brief moment in December 1882, Robert E. Lee’s son legally owned Arlington National Cemetery, headstones, monuments, and all. It could not last, and everyone knew it. Thousands of Union dead were already buried in that ground, and there was no realistic scenario in which the Lee family could evict the United States military from its national cemetery.
So in 1883, Custis Lee agreed to sell the estate to the government for $150,000, a huge sum at the time, and walked away from the case legally victorious and practically defeated in the longest lawsuit connected to the entire Civil War. He never lived at Arlington again. He had simply proved, decades after the fact, that the government had taken it from his family illegally in the first place, and forced them to pay to keep it. Custis never married either, a fact rarely mentioned alongside his sisters’ unmarried lives, but just as striking.
He spent nearly all of his remaining decades serving an institution and a name that were not entirely his own, running a university built around his father’s memory, sleeping in rooms a short distance from the chapel where his father was eventually buried. Where his sisters scattered across continents to escape the weight of the family name, Custis built his entire adult identity around carrying it. While Custis fought in the courtrooms, his younger brother Rooney was fighting to rebuild something closer to home. His path was the most brutal of all the Lee siblings.
Captured by Union forces in 1863 while recovering from a battle wound, Rooney spent months as a prisoner in a Northern jail, and the family’s desperate desire to get him back produced one of the most extraordinary offers of the entire war. Custis, still serving in the Confederate army, volunteered to take his brother’s place so Rooney could be released to see his dying wife. The Union refused the offer outright. Rooney stayed imprisoned.
Charlotte died of what family letters described as severe anxiety and exhaustion brought on by his capture, and Rooney could not get home in time to say goodbye. He returned from the war to find his plantation, White House, burned to the ground, and both of his young children dead. Most people would have understood if he simply gave up. Instead, Rooney rebuilt the plantation from the ashes, remarried, started a new family, and slowly worked his way into public life, first as a state senator in Virginia, and eventually, by 1887, as an elected member of the United States Congress, representing the same district his father had once tried to tear away from the nation Rooney now helped govern.
He served until his death in 1891. He was the only son of Lee to hold federal office in the government the Confederacy had fought against. The youngest son, Robert E. Lee Jr.
, chose the quietest path of all. He inherited Romancoke, another property that had belonged to his mother’s family, and spent the rest of his life there as a farmer, out of the spotlight compared to his brothers. Decades later, in 1904, he published a memoir of his father, drawn from personal memories and letters, offering a rare and intimate window into what life at Arlington was like before the war tore it apart. He died a decade after the book appeared, the last of Robert E.
Lee’s sons to go, having spent his postwar life working the same land his family had lost everything to hold onto. So what really happened to Robert E. Lee’s family after the fall of the Confederacy? There was no single fate, but eight completely different ones, scattered among courtrooms, plantations, European cities, a seat in Congress, and at least one police station in Alexandria.
A mother who never returned home. A daughter who did not live to see peace. Two more daughters who chose exile over roots, one of whom accidentally made headlines forty years after the surrender. A son who defeated the United States government in its own Supreme Court and still lost.
Another who rebuilt from the ashes literally and ended up serving in the very Congress his father had resigned from in 1861. And the youngest son who was content to stay on the land and write everything down before it faded into memory. None of them got Arlington back. None of them got their old lives back either.
Instead, they got nearly forty more years of a war that was technically already over. They fought it this time through courtrooms, through silence, through distance, and through the slow, painstaking effort of figuring out who they were after the uniform, the plantation, and the world that produced both had disappeared. What makes the Lee family’s story worth telling is not that they lost a war. Many families lost that war.
It is that their loss settled nothing for them in the way a surrender is supposed to settle a story. There was no clear ending waiting for them after Appomattox, no return to the world they had known, only decades of scattered individual attempts to build lives from what remained. Some tried to reclaim it through the courts. Some tried to escape it across the oceans.
Some simply stayed on what was left of the land and quietly held the family together. None of it looked like victory, and none of it looked like the defeat people usually imagine. It looked like eight different people spending the rest of their lives trying to figure out what to do with a name that had been carved into history before any of them had the chance to decide what it meant.
