Morris Offit loves watching people enter the offices of his New York wealth-management firm, Offit Capital Advisors, which is filled with antique American flags. "People are overwhelmed," he says. The flags get people talking—and, in some cases, buying their own.

Mr. Offit took up his hobby in 2004 when he accompanied his wife, Nancy, on one of her regular prowls through an antiques show. A small antique flag at the booth of Jeff R. Bridgman caught his eye, and Mrs. Offit encouraged him to buy it. Before long, he'd made friends with Mr. Bridgman, a York County, Pa., dealer who specializes in "early American Stars and Stripes," and a collection was born.

Now, the 74-year-old Mr. Offit owns 40 to 50 flags and other historical textiles, including one of the earliest known kerchiefs depicting George Washington. Among his favorites: an 1876 flag from the U.S. centennial celebrations that hangs in his office.

In the realm of antiques, historical American flags and political textiles form a small category, but one with truly passionate collectors. The rare pieces dated before the 1820s fetch the highest prices. In 2006, Sotheby's set the record when it sold a Revolutionary War cavalry flag for $12.33 million, well over its $3.5 million high estimate. In the same sale, the same buyer, who has remained anonymous, purchased three additional Revolutionary War battle flags for $5.05 million.

Mr. Offit says such purchases are beyond his flag budget. This pastime "has all to do with decoration, history and patriotism. It's a good warm feeling to collect American flags," he says.

Owning a piece of history is one of the common feelings that drives flag collectors, says David N. Redden, a Sotheby's vice chairman who handles flags. The very best specimens, he says, "are magical objects, combined with an incredibly rich story." Many designs are also beautiful, he adds. Mr. Offit says it's the design that attracts him. "What most people don't realize," he says, "is that before 1912, there was no official configuration" of the U.S. flag. "So there were flags with medallions and great star patterns."

In 1777, Congress decreed: "Resolved that the flag of the United States be 13 stripes alternate red and white, that the union be 13 stars white in a blue field representing a new constellation." In 1818, Congress set the number of stripes as 13, but let the stars grow with each state.

President William Howard Taft determined to set an official arrangement, and people vied to be the designer. Wayne Whipple of Philadelphia devised a configuration placing 13 stars, for the original colonies, at the center arranged in a six-point star, which is surrounded by a circle of 25 stars representing the states that had joined the Union by the centennial in 1876. Around that is a wreath of 10 stars, widely spaced, for the states that had joined since then.

Taft rejected the Whipple design, but Mr. Bridgman acquired a specimen of it from Whipple's descendants and sold it to Mr. Offit. It now occupies pride of place in his firm's foyer.

Offit flags and textiles include one from Lincoln's 1860 campaign

A crib quilt

A banner with Whipple's rejected design