Broom was born in 1752 at Wilmington, DE., the eldest son of a blacksmith who prospered in farming. The youth was educated at home and probably at the local Old Academy. Although he followed his father into farming and also studied surveying, he was to make his career primarily in mercantile pursuits, including shipping and the import trade, and in real estate. In 1773 he married Rachel Pierce, who bore eight children.