Delta Apparel is to close its textile manufacturing facility in Fayetteville, North Carolina, with the loss of more than 100 jobs.