Fashion chain Casual Male Retail Group Inc on Thursday posted a big jump in its fourth quarter net loss as it was hit by restructuring charges related to its exit of its Levi's/Dockers and Candies businesses.