US clothing company Perry Ellis swung to a second-quarter loss as the as earnings were hit by costs related to its restructuring and early retirement initiatives.