The secret history of the Hampton’s Sylvester Manor

Landscape historian Mac Griswold unveils the truth behind a Shelter Island slave plantation in her new book The Manor: Three Centuries at a Slave Plantation on Long Island.

andrew bush/courtesy of farrar, Straus and giroux
Mystery Manor Built in the high style of early-18th-century Newport, Sylvester Manor is a repository of American history from Colonial times to the present day.

HC&G: When did you first become interested in landscape history?
Mac Griswold, garden historian: I started as a child, when I was forced to walk an hour every day for my health. I grew up in the great estates area of New Jersey and would create stories while walking in my neighborhood. By the time I was 35, garden history was just being established as a discipline, and I attended a special program at Radcliffe to learn about different cultures through the study of landscapes and gardens.

Sylvester Manor is the subject of your brand-new book, The Manor: Three Centuries at a Slave Plantation on Long Island [Farrar, Straus and Giroux]. What compelled you to write about it?
I was rowing around Shelter Island one day with a friend, and after paddling up a creek we found ourselves in the 18th century. We came upon a great old house with 12-foot-tall boxwoods, and I knew it was a real garden. We were trespassing, saying “Yoo-hoo,” but no one was there.

How did you gain access?
I wrote to the owners, Andrew and Alice Fiske, several times, and finally Andy Fiske asked me to visit. In 1984, I was invited to see a great beech tree planted by the botanist Asa Gray in the 19th century.

What else did you find?
Andy showed me two other startling things: a set of stairs that went to the slave quarters— he said slaves had slept in the attic—and a hidden door cut into a wallpapered wall, which concealed a closet with family papers dating from as far back as 1651, when his ancestors first settled here.

courtesy of farrar, Straus and giroux
Land Of Discovery (clockwise from top left) Dating from the early 1900s, Cornelia Horsford’s Colonial Revival–style rose trellises have a giant yew as a backdrop.  Julia Dyd Havens Johnson (1809–1907) worked as a housekeeper at Sylvester Manor and owned 21 acres of land overlooking Dering Harbor. A 20th-century garden folly stands beyond an overgrown boxwood maze and towering yews. Professor Eben Norton Horsford enjoys an afternoon with family and friends near Gardiners Creek.


When was the manor house built?
In 1737, by founder Nathaniel Sylvester’s grandson, Brinley Sylvester. It’s the earliest Georgian house on Long Island. Brinley was an elegant young man raised in Newport, where houses like this were common. He had won
a lawsuit winning back 1,000 acres of the family’s original 8,000 acres, tore down an old Jacobean house, and built the manor. Since then, the parlor has had only two coats of paint. The atmosphere of the house is mysterious, secretive. It’s a house with ghosts.

What part did the Sylvesters play in the slave trade?
Slavery in New York began in the Colonial era and lasted until July 4, 1827; slavery at Sylvester Manor ended in 1820. The main basis of the Sylvesters’ wealth during this time was in the slave trade. Brinley Sylvester was the port collector of Southampton and took a percentage on the four pounds levied on every imported African. He probably had as many as 15 slaves himself.

Have you been able to trace any of the slave families?  
The trail stops in the 1870s, with freed slaves Elimus Derby and his wife in Sag Harbor. After slavery, many former slaves became tenant farmers on the property.

How did the gardens evolve?
In the 19th century, Eben Norton Horsford, a chemist who invented baking soda, married into the family, and the gardens became like a beloved old aunt. They used plants popular in the Victorian era, such as celosia, amaranth, and lotus. In the mid-20th century, Andrew Fiske married Alice Hench, a natural-gas heiress who was a great gardener and very interested in preserving the family history. Together they kept the place intact and made it even more beautiful.

Who owns the property now?
Eben Ostby, a founding member of Pixar Animation Studios, who has given papers about Sylvester Manor to New York University for public use. The property now operates as the Sylvester Manor Educational Farm, a nonprofit dedicated to agriculture and the arts. Bennett Konesni, a musician and farmer, is the 15th member of the Sylvester family to steward the place, parts of which are coming back to life as an organic farm, with young farmers living on the property and even running a popular farmstand there.