While the work was fiction, and the existence of the well-worn letter "A" found in the Salem custom-house was certainly a literary device, Hawthorne's famous story did come from the experiences of his ancestors.
From Today in Literature:
Among his seventeenth-century ancestors were two sisters who had been forced to sit in the Salem meetinghouse wearing forehead bands identifying their incestuous conduct (while their brother hid out in Maine). The Scarlet Letter also came from Hawthorne's general guilt over the Puritan enthusiasms of some of his other ancestors -- one had been a judge at the witch trials -- and his feeling that his hometown was a place of gloom and convention.King continues
Hawthorne's attempts to escape Salem included a short stay at Brook Farm, the Transcendentalists' utopian community outside of Boston. Although at first invigorated by the new thinking and fresh air, he soon found himself permanently volunteered to the manure pile, and reappraising town-life: "a man's soul may be buried and perish under a dung-heap, or in a furrow of the field, just as well as under a pile of money."So Hawthorne returned to Salem and his work at the customs-house while continuing to hope for the time when he would be able to make a living from his writing. Not until he was fired from his job in the summer of 1849 did he make another attempt at the tale which had been previously been know as "Endicott and the Red Cross" about "a young woman with no mean share of beauty, whose doom it was to wear the letter A on the breast of her gown."
Despite his predictions that it would "weary very many and disgust some," The Scarlet Letter was immediately popular, allowing Hawthorne to move away from Salem with this good riddance: "I detest this town so much that I hate to go into the streets or to have the people see me. Anywhere else, I shall at once be entirely another man."