Clothing designer Perry Ellis International Inc on Wednesday said it swung to a second quarter loss as it was hurt by high administrative costs related to its recent acquisition of Salant Corp.