The door of the Fifth Avenue mansion swung open, and beyond the narrow vestibule the house released its visitors into a double-height entrance hall. Before them a grand bifurcated staircase descended beneath a stacked row of columns, and at the top of that staircase light spilled from behind strange greenery. Behind an ornate wrought-iron railing, a winter garden glowed under a series of skylights, and inside it the soft sound of water trickled slowly down a marble wall. This was only the first wonder the house had to offer.

It belonged to James Speyer, a man born into wealth so old and so vast that rumors in New York society claimed his family had surpassed the legendary Rothschilds themselves. James Speyer was born in New York City in 1861, heir to a banking dynasty that had roots deep in Frankfurt, Germany. The family fortune began with Joseph Lazard Speyer, who took over the management of the Bank of Elissen in 1818 and renamed it J. L.
Speyer-Elissen. His sons carried the banking empire across Europe and into the United States. James’s uncle, Philip Speyer, had emigrated to America in 1837 and established the family bank’s American branch, which would become Speyer & Co. From its early days the firm was a powerful institution, playing a decisive role in financing railroads and public infrastructure across the growing nation.
Although born in New York, James spent his formative years in Germany, where he was raised in his family’s Jewish faith and prepared for a life in banking. He trained in the family branches in Frankfurt, Paris, and London, and in 1885 he returned to New York to join Speyer & Co. By 1899 he had taken his place as head of the firm. Under his direction the company grew into the third largest investment banking house in America; by 1913 it managed the staggering sum of 2.
4 billion dollars, a figure equivalent to more than 76 billion dollars today. In 1897 James married Ellen Prince Lowery, a Christian. Among the elite of New York the union stirred whispers, and some social doors remained closed to him because of religious prejudice. Yet by all accounts the mixed marriage was a happy and productive one.
Together the couple pursued not only the accumulation of wealth but also meaningful philanthropy and social influence. They played an essential part in founding the Museum of the City of New York, opened hospitals for animals, and established scholarship programs, among dozens of other causes. By 1911 James decided to take his place among the wealthiest members of New York’s high society by building a home along Fifth Avenue. He commissioned the celebrated architect Horace Trumbauer to design the residence.
But it was Trumbauer’s chief designer, Julian Abele, one of the first professionally trained African American architects in the country, who brought the mansion to life with a stunning design in the style of Louis XIV. The interiors, designed by Ogden Codman Jr. , followed the same opulent Louis XIV manner, with enameled panels, gilded touches, and muraled walls. The house was a true palace, and it held one of the largest private art collections in America, including works by Rembrandt.
Inside, the journey began at the front door. Crossing the narrow vestibule, guests were released into the double-height entrance hall. The grand bifurcated staircase descended the space beneath a stacked row of columns. Most visitors would ordinarily explore the main floor first, but what lay at the top of the stairs was simply too tempting to wait for.
At the second-floor landing, light poured in from behind exotic plants. Behind a wrought-iron railing, natural light flooded a winter garden through a series of skylights. Entering this hall, one heard the sound of water slowly trickling down the marble wall behind the foliage. It was the first marvel of the house, but far from the last.
Taking the back staircase down to the main floor, the tour of the public rooms began. First came the dining room, finished from floor to ceiling in wood paneling interrupted by marble columns. High above, a gilded cartouche surrounded a church mural painted in bright, vivid colors. To the left of where one entered the dining room, doors on the right led into the white-and-gold drawing room, which seemed almost to glow even though no lights were visible.
The ambient lighting was hidden away in the folds of the millwork and above the ornate cornice. In stark contrast, the hall was covered entirely in rich wood paneling, with a dazzling electric crystal chandelier hanging from the cove ceiling. It was worth pausing to appreciate the intricate woodwork: the fluting on the columns, the continuous wooden vine wrapping the frieze, the flat surfaces above every doorway. From there the tour continued into the library, which featured hand-painted ceiling beams and an oversized stone fireplace.
From the rustic chandelier to the European furnishings, the library was intended to imitate the rooms found in castles and country houses centuries old. Every room so far had been enormous, but the largest had not yet been seen. Returning up the stairs and crossing the winter garden once more, one rounded the other side to find the ballroom. Its gilded wall panels and polished wooden floors reflected the light from four crystal chandeliers and eight additional wall sconces with dazzling effect.
Gleaming mirrors were set into the paneling throughout the room, making the space appear even larger than it truly was. In 1921 tragedy struck. Ellen died after a long struggle with illness. Her death marked a profound turning point in James’s life.
In the years that followed, he devoted himself to preserving her legacy and the causes they had championed together. He suffered personal and financial setbacks, including a destructive fire that destroyed priceless works of art in his palace. Yet Speyer remained a respected figure in New York society. He retired in 1938, choosing to close Speyer & Co.
rather than leave his name in the hands of others. Then, in 1941, he died in his beloved mansion on Fifth Avenue. His will bequeathed much of his fortune to charity and left his palace to the Museum of the City of New York. Against his wishes, the museum sold the mansion, and it was eventually demolished to make way for new development.
Although the palace no longer stands, the legacy of James Speyer continues through the institutions he supported and the lasting influence of his philanthropic endeavors. The winter garden is gone, the gilded ballroom reduced to rubble, the Rembrandts scattered to wherever his collections were dispersed. What remains is the record of what was built, and the memory of a family that, according to the whispers of its own time, may have been richer than the Rothschilds themselves.