Apparel business Perry Ellis suffered a drop in fourth quarter profit, despite a double-digit revenue increase, after a difficult holiday season.