On the north side of the creek where Market Street now runs was Lot No. 80 of the Grand Modell, containing a little more than nine acres, granted in 1681 to Sir Peter Colleton. It was sold in 1736 by his grandson, the Hon. John Colleton, to George Hunter, Charles Pinckney and Thomas Ellery. By 1739 the tract had been laid out with streets called Charles (now part of Anson), Pinckney, Thomas (now North Market) and Ellery (now Guignard). In 1746, Charles Pinckney, Chief Justice of South Carolina, built his fine brick mansion on the Bay in Colleton Square. During the Pinckney family's absence in England, 1753-58, the mansion was occupied by Gov. James Glen, and became known as the "Governor House," and the bridge over the canal, at the mouth of the creek, which bridge connected Colleton Square with the main part of town, at East Bay, was called the "Governor's Bridge." The mansion, which had Ionic pilasters and a pediment on the facade, remained in the Pinckney family until it was destroyed by the great fire of 1861.
Charles Fraser, in his Reminiscences, stated that the Charleston Orphan House was located in Colleton Square, on Church Street north of the creek, before the Orphan House on Calhoun Street was constructed. (McCrady, 2:398. Ravenel, Charleston, The Place and the People, 141, 156. Whitelaw & Levkoff, 17. Rogers, Charleston in the Age of the Pinckneys, 120-121. Smith & Smith, Dwelling Houses, 269-270. Fraser, 28.)