Delta Apparel might have posted a strong set of full-year financial results, but it admits that it still has work to do on improving the Soffe brand.