Apparel maker Perry Ellis International has swung to a fourth quarter loss after writing down the value of some of its brands and taking higher markdowns at department and specialty stores.