Apparel and shoes chain J Crew said three directors plus its co-founder are leaving the company.