The Hazardous Adventures of the Refugees of 1776 - II
#4 in East Hampton and the Revolution — A Series
Long Island Sound, detail of painting by Archibald Cary Smith (1837-1911)
PART II: Exile, Occupation, and Return
Introduction: A torch on fire at both ends
In Part I of my mother's refugee research, she explored how 1,124 Long Island residents evacuated across the Sound to Connecticut in the chaotic weeks following Washington's withdrawal from Long Island. We learned about the logistics—the captains, the bills, the wagon loads, the livestock crammed into ship holds. Urgency required in the face of imminent British occupation.
But Part II confronts the harder questions: What happened next? How do you live as a refugee for seven years? What choices do you make returning home, and what do those choices reveal about who you were and what you valued?
My mother was fascinated by the individual family stories that emerged from the refugee claims and pension applications. She tracked Aaron Isaacs making six separate trips across the Sound with his merchandise, finally returning to his "charming house" in 1782 after six years in exile. She documented Colonel Abraham Gardiner's dramatic change of heart—on September 7, 1776, he forced his neighbor Colonel David Mulford to swear allegiance to King George III causing Mulford to flee to Connecticut the next day, the day after that, Gardiner himself became a refugee, having "thought better of his situation."
Imagine the extreme pressure, the permanence of the consequences, the wrenching personal choices.
Colonel David Mulford was able to return to East Hampton, only to die of smallpox in 1778. Town Supervisor Burnett Miller remained in the Hudson Valley with the New York State government, choosing public service over home. Captain Thomas Wickham also stayed in the Hudson Valley, his political connections more valuable than his saltbox house on Main Street.
My mother was also drawn to the experience of those who stayed behind under British occupation. She found and quoted Reverend Samuel Buell's devastating letter from September 22, 1776:
…the people are as a torch on fire at both ends…..
This revealed the impossible position of non-combatants: punished by both sides, caught between armies, watching their property stolen or destroyed regardless of their loyalty.
The whale-boat warfare she describes—both Loyalists and Patriots raiding the coast, "never questioning the loyalty of the people they were robbing"—reveals the chaos and moral collapse that occupation brought. The destruction of John Brown's house on Fisher's Island shows the systematic devastation of the region's economy.
By the time the truce was announced and refugees began returning, East Hampton had been fundamentally changed. Some houses stood empty. New people had arrived—like William Paine, an educator, who would establish schools. The fourth minister, Lyman Beecher, purchased a vacant house and spent considerable money repairing it.
This is the long aftermath of the Battle of Long Island: not just seven years of occupation, but the permanent reshaping of a community.
My mother understood that wars don't end when the fighting stops. They end when the last refugee decides whether to return home, and what that decision costs them.
The Hazardous Adventures of the Refugees of 1776
#4 in East Hampton and the Revolution — A Series
By Sherrill Foster, East Hampton Town Historian 2003-2007
A series based on Around the Green columns, updated by Mary Foster Morgan, 2026.
PART II: Exile, Occupation, and Return
Aaron Isaacs: a Merchant's Calculation
As news of the occupation spread, Aaron Isaacs of East Hampton immediately moved his merchandise to East Haddam in six boatloads of 'effects and goods'. Listed were 56 sides of leather and 20 calves skins. His livestock included a yoke of oxen, 14 cattle and horses and 3 hogs. He had wagon loads of hay, corn; 15 bushels of oats, as well as a 'riding chair', the luxury item for individual travel.
With his wife and eight children he moved into East Haddam, where one of his merchant friends had warehouses. This was not a panicked flight but a calculated business decision. Isaacs had the commercial connections and portable assets that allowed him to relocate his entire operation.
Isaacs would return to East Hampton as early as 10 June 1779 to get a horse and a quantity of flax back to East Haddam. Even during the occupation, he was managing his property and inventory, making strategic trips to retrieve valuable goods. But he could not return to his charming house until 12 October 1782, when the truce was effected—more than six years after he first fled.
Burnett Miller: Public Service Over Home
Burnet Miller, Supervisor of the Town of East Hampton from 1746-1777 and a delegate to the 3rd and 4th Provisional Congresses, was soon in Stonington, having loaded his effects onto the boats of seven different Captains to take him and his family of ten with four enslaved persons across the Sound.
Miller's evacuation reveals his status and resources—seven different captains were required to transport his household, which included four enslaved. But Miller would never return to East Hampton. He was a member of the New York Assembly held in Kingston, and continuing his government service, remained in the Hudson Valley.
This was a choice, but also a recognition of political reality. Miller's value to the Revolutionary cause was in governance, not in reclaiming his Main Street house. His prominence made him useful in exile, and that usefulness became a new life.
Thomas Wickham: an Auditor's Position
Thomas Wickham left the new Town dock at Sag Harbor on September 7, 1776 in seven vessels for Stonington with his family of eight and his three enslaved workers. Like Miller, Wickham's evacuation required multiple vessels, revealing both his wealth and the extent of his household.
Wickham was appointed, together with the refugees Thomas Dering and John Foster, to be an Auditor of the Claims for payment for the cross Sound transportation. The offices were in Middletown, on the Connecticut River.
This appointment was significant. Wickham and his fellow auditors would process all the refugee claims—the bills from captains, the receipts for storage, the accounting of lost property. They were literally the gatekeepers who determined whose losses would be compensated and at what rate. This was considerable power, and it created a new role for Wickham.
Like Miller, Wickham moved into the Hudson Valley with the new New York State government. He never returned to his saltbox house across from the 1770 House. Political connections and administrative expertise had become more valuable than home.
Mulford Family: Coercion and Consequence
On September 7, 1776—the same day Thomas Wickham evacuated—Colonel Abraham Gardiner and a company of redcoats surrounded the residence of Colonel David Mulford, and forced him to swear the oath of allegiance to King George III! Not a request, but an armed coercion.
Gardiner, backed by British soldiers, compelled his neighbor to publicly declare loyalty to the King. It was an act of intimidation designed to secure Mulford's cooperation with the occupation. Colonel Mulford was an important Revolutionary leader.
The next day, Colonel David Mulford left for Stonington, taking at least one of his eight enslaved people, a man named Jack, and some 13 head of cattle. His son—David Mulford Jr., age twenty-two, a weaver who held the rank of Major—initially remained in the house with several of the family's other enslaved workers, most likely to secure the property and household.
Colonel Mulford's response to the forced oath was immediate flight—as an important Revolutionary leader, he was needed in Connecticut where New York State's government-in-exile had established operations in Middletown to coordinate the refugee crisis and military efforts. He could serve the cause better there than living under British occupation on Long Island.
Whether David Jr. stayed throughout the occupation or later joined his father in Connecticut as a refugee remains unclear from the surviving records. What we do know is that he eventually returned to the family house, continued his weaving trade, and after the war married Rachel Gardiner—the daughter of the very man who had forced his father to swear the oath of allegiance. They raised four children in the Mulford house.
Colonel David Mulford was able to return to East Hampton, only to die of smallpox in 1778. His widow returned after the war.
The marriage of Rachel and David, connecting the Gardiner and Mulford families despite the coercion and conflict of September 1776, suggests something about how East Hampton had to rebuild itself after the war. Old grievances had to be set aside. Marriages had to bridge the divisions the war had created.
Colonel Abraham Gardiner's Change of Heart
On September 9th, two days after forcing Colonel David Mulford to swear allegiance to King George III, Colonel Abraham Gardiner himself fled to Connecticut.
This dramatic reversal suggests either that Gardiner had been acting under pressure himself, or that he quickly realized the British occupation would be intolerable regardless of his declared loyalty.
Captain James Harris of Connecticut moved Gardiner's effects to Saybrook where Harris had a wharf and store. Gardiner is listed with seven in his family: his wife Mary; his two teenage sons, Nathaniel (who at 17 soon became a Surgeon's mate in the Hospital Service) and Abraham; his daughter Rachel, already 25 but betrothed to Colonel David Mulford's son David, the weaver who remained in his family home; and two of Gardiner's five enslaved help. He brought his mare and provisions. Four captains' sloops were needed for all his goods.
The betrothal of Rachel and David would survive the war.
Life under Occupation
Colonel Abraham Gardiner's large and elegant house on Main Street became vacant when he fled on September 9th. It became the British military headquarters for the seven years of the occupation. Governor Tryon, Sir William Erskine and Sir Henry Clinton are mentioned as occupying the house, along with Naval officers.
Gardiner's "mansion house" was the finest residence in East Hampton—of course it became headquarters. The British needed administrative centers, officer housing, and command posts throughout occupied Long Island. The most prominent houses were obvious choices.
15 Houses Vacant
The British took a census in the early spring of 1777, which showed 15 houses vacant in East Hampton. Among those vacant was that of Colonel Abraham Gardiner, [now the LVIS headquarters], the Aaron Isaacs house [now the Home Sweet Home museum], the Burnett Miller house, [now the 1770 house and restaurant], and the Thomas Wickham house [a private residence across the street from the 1770 House].
Fifteen vacant houses in a town of 1,250 people represent a significant portion of the most substantial dwellings. These weren't modest cottages—these were the large houses on Main Street, the homes of merchants, town supervisors, militia colonels. The leadership class had fled.
A Torch on Fire at Both Ends
Both those who fled to Connecticut and those who remained had difficult times. Possessions were stolen or destroyed, the non-combatants on Long Island were treated miserably; the refugees in Connecticut were often robbed of their possessions.
The Reverend Buell wrote on 22 September 1776 to Governor Trumbell of Connecticut:
…the people are as a torch on fire at both ends, which will be speedily consumed; for the continental whigs carry off their stock and produce, and the British punish them for allowing it to go. I hope the Whigs will not oppress the oppressed, but let the stock alone.
Buell's metaphor is devastating. The people are burning from both ends—the patriots seizing livestock and produce to deny it to the British, the British punishing civilians for allowing that seizure. The non-combatants were trapped in an impossible position.
Buell's plea—"I hope the Whigs will not oppress the oppressed"—suggests that patriot forces were treating Long Island residents harshly, confiscating property with little regard for the vulnerability of those left behind. This was not a clean division between loyalist and patriot. This was chaos, with civilians suffering regardless of their political sympathies—provisioning British troops because they had no choice, losing livestock to patriot raids, compensated by neither side.
In East Hampton, The Huntting Inn became a neutral haven during British occupation, where officers met. Local histories show that descendants of Rev. Nathaniel Huntting (1675-1753)—including Nathaniel (1702-1770) and his son Nathaniel (1730-1801)—avoided enlistment in patriot militias, Continental forces, or loyalist units, managing the inn as a way to protect their property. This neutral stance was similar to Rev. Buell's decision to stay "to protect his flock." Both chose pragmatic neutrality, knowing that most East Hampton residents supported the patriot cause.
700 in Southampton, 100 in Sag Harbor
In February 1779, 14 companies (700 men) were quartered in Southampton. In April 100 men and 2 field pieces were in Sag Harbor. There seems to be no information about how many men were quartered in East Hampton.
The presence of 700 men in Southampton gives a sense of the military occupation's scale. These troops needed housing, provisions, fuel, fodder for horses. They requisitioned what they needed from the local population. The civilians who remained had to house, feed, and accommodate an occupying army for years.
Whale-boat Warfare and Raids
In May 1777, Sag Harbor experienced some of the whaleboat warfare so common along the Sound shore. Both Loyalists and Patriots engaged in these raids. Coming ashore they took what was there, never questioning the loyalty of the people they were robbing. The 30-foot whaleboats were moved by 2 to 10 pairs of oars. The patriots' object was to procure food for their army, and then to destroy what they could not carry, so the enemy could not get it.
Was this principled military action or opportunistic raiding by both sides? The whaleboats were fast, maneuverable, and could land anywhere along the coast. The raiders took whatever was available—livestock, grain, stored goods—without regard for whether they were robbing patriots or loyalists.
For the civilian population, this meant that political loyalty offered no protection. You could be a committed patriot and still have your property seized by patriot raiders. You could be a reluctant loyalist and still be robbed by loyalist forces. The only certainty was vulnerability.
Fisher's Island, Raided
A report in March 1777 details the destruction by the British occupying army. John Brown owned Fishers Island, raising produce on it for the West Indies trade. The British Fleet of 20 ships at anchor in Gardiner's Bay went to Fishers Island and took 106 sheep, 8 oxen, 11 cows, 22 yearlings, 26 swine, 24 turkeys, 48 fowls, 123 bushels of corn, 100 bushels of potatoes, 5 1/2 tons of pressed hay and 3 cords of wood. They also took a barrel of pork out of the cellar, blankets, sheets, and shot some sheep. They paid for most of the stock.
However in July 1779, the British put powder in the cellar and blew up the house which had been lately improved by John Brown. They fired the surrounding barns and other buildings.
The meticulous inventory—106 sheep, 8 oxen, 22 yearlings, 123 bushels of corn—reveals the systematic nature of the confiscation. This was not random looting but an organized seizure of resources by a military force provisioning itself from the occupied territory.
The initial raid in March 1777 even included payment for most of the stock. The British were maintaining some pretense of legitimate requisition rather than theft. But by July 1779, that pretense had collapsed. They blew up the house with powder in the cellar and burned the barns. This was punitive destruction, designed to deny resources and break the will of the civilian population.
John Brown's Fisher's Island operation represented the West Indies trade that had made East Hampton prosperous. Its methodical destruction symbolized the complete disruption of the regional economy.
John Brown's Fisher's Island operation represented the West Indies trade that had made East Hampton prosperous. Its methodical destruction symbolized the complete disruption of the regional economy.
Textiles: The Currency of Survival
Textiles were essential to Revolutionary-era households—not just for clothing, but as a form of portable wealth that could be traded for food, shelter, and hard currency. Mather's Refugees documents lists of items that those living in Connecticut wanted from occupied Long Island. Most were textiles: lengths of cloth, finished garments, buttons, thread, ribbon. These could be sold in Connecticut for hard cash—something refugees desperately needed but found difficult to obtain.
The demand for textiles reflected both practical necessity and economic opportunity. Refugees who had fled with little money could trade cloth goods for essentials. Connecticut residents, cut off from their usual supply chains, were willing to pay premium prices.
Long Islanders in Connecticut who wanted to retrieve their possessions could obtain a permit from the Governor in 1779. With this official warrant, they could hire a boat, sail to Long Island, collect their belongings, and return. Each permit specified what items could be transported.
But even these official warrants didn't prevent disputes over what constituted legitimate refugee property versus illegal contraband. Goods stolen from loyalists or seized from occupied homes were being smuggled into Connecticut shore towns for sale, mixed in with legitimate refugee possessions.
To prevent smuggling and unauthorized trade, boats traveling to Long Island were inspected by qualified personnel before departure. Lists were made of what refugees claimed they intended to retrieve. Upon return, the boats were inspected again to verify the cargo matched the approved list.
Despite these precautions, authorities still attempted to seize goods they deemed suspicious—even from refugees with proper permits. The problem was definitional: what looked like commercial stock to inspectors might be a merchant's legitimate business inventory or valuable household goods.
A notable case involved Richard Seamen who brought over substantial quantities of commercial goods along with household items: yard goods (cloth sold by the yard), buttons, sealing wax, "sowing thread" (sewing thread), tape, ribbon, buckles, horn combs, and leather gloves. These weren't just household necessities—this was merchant stock.
The case went to a Connecticut court on suspicion that these were illicit goods intended for resale rather than personal refugee property. Seamen defended himself by stating he "was in trade" on Long Island—in other words, he was a merchant whose business inventory happened to be portable and valuable. He successfully argued his case and was permitted to sell his commercial stock in Connecticut.
The British occupation had created an economic crisis where normal trade routes were severed, legitimate businesses were disrupted. The occupation also opened up a legal crisis: Was this a refugee fleeing with valuable possessions, a merchant conducting legitimate trade, or someone profiting from stolen goods? Refugees needed to survive, merchants needed to continue business. Smugglers exploited the chaos and the authorities couldn't tell them apart. Textiles, being both necessary and valuable, sat at the center of this moral and legal confusion.
Return and Slow Reoccupation
At the news of the Truce [preliminary articles of truce agreed to November 30, 1782, Treaty of Paris, signed September 3, 1783, ratification not until 1784] Long Island residents started to return, always with the notice of authorities. The men first came back with lumber to rebuild their houses, horses to work the fields, and then their families and furniture. The return was not instantaneous. It was a careful, staged process. Men came first, surveying damage, beginning repairs. They brought lumber because buildings needed reconstruction. They brought horses because farms couldn't be worked without them. Only after these preparations did families and furniture follow.
The requirement to notify authorities suggests the New York State government wanted to track and manage the reoccupation. After seven years of occupation, property rights were unclear, houses had been damaged or destroyed, and the returning population needed some coordination to avoid chaos.
Vacant Houses – Who Returned, Who Didn’t
Some owners decided not to return to East Hampton.
This simple sentence contains enormous human consequence. After seven years in exile, some refugees had built new lives. Their children had grown up in Connecticut or the Hudson Valley. They had new homes, new occupations, new communities.
Burnett Miller was a member of the New York Assembly held in Kingston, and continuing his government service, remained in the Hudson Valley. Captain Thomas Wickham moved into the Hudson Valley with the new New York State government.
Miller and Wickham had found new roles in the Revolutionary government. Their political service had become their identity. Return would have meant giving up positions of influence and importance.
Colonel Abraham Gardiner returned to his large house on Main Street, which was inherited by his medical son Nathaniel. His daughter Rachel did marry Colonel David Mulford’s son, David Jr., the weaver who had held the rank of Major, and they raised four children in the Mulford house. David’s marriage to Rachel—after the forced swearing oath incident of their two fathers —suggests a deliberate choice to bridge any divisions and rebuild the community.
When Gardiner reclaimed his mansion house after seven years of British occupation, one wonders what condition it was in after serving as military headquarters. His son Nathaniel, who had served as a surgeon's mate in the Hospital Service at age seventeen, inherited the house—a reward, perhaps, for his service.
Aaron Isaacs returned to East Hampton, bringing with him his new son-in-law, William Paine, an educator and teacher who would soon be conducting schools.
Isaacs' return brought new expertise to East Hampton. William Paine represented the kind of expertise the town would need to rebuild. Education had been disrupted by the war; Paine would help restore it.
Other vacated houses remained empty. When Lyman Beecher became the fourth minister in East Hampton, he purchased one and spent time and money repairing it to habitable form. Others fell into disrepair.
Not all the prominent families returned. Their houses stood empty, slowly deteriorating. Lyman Beecher's purchase and repair of one vacant house suggests they were sold at low prices to anyone willing to invest in restoration. But some houses simply fell apart, their owners dead or permanently relocated, no buyers willing to take on the expense of repair.
Whaling and New Prosperity
New whaling techniques brought renewed income to the area. Deep-sea whaling voyages replaced the unprofitable shore whaling of the colonial period. By the 1810s people were building new large Federal-style houses, many of which line Main Street today.
Clinton Academy opened for classes January 1, 1785, a coeducational academy, built with funds contributed by local citizens at the request of Reverend Samuel Buell, pastor of the East Hampton Presbyterian Church. It was one of the first institutions of its kind, chartered by the New York State Board of Regents, and was named after Governor George Clinton who contributed the original bell that hung in the building's cupola.
The economic recovery took decades. Those Federal-style houses were built by a different generation—the children who had crossed the Sound as evacuees in 1776. Their parents had borne the cost of independence: homes they never reclaimed, neighbors who never returned, and a community that could never quite be what it was before the war.
Sources:
For more information on these events and the soldiers and sailors named, see Frederic Mather, Refugees of 1776 from Long Island to Connecticut (Albany 1913, reprint Genealogical Publishing Co. 1972, and Clearfield Company 1995).