Apparel maker Perry Ellis International Inc is exiting the men's specialty store business and its Dockers outerwear license in a bid to save up to $8m.