Specialist clothing retailer Casual Male Retail Group saw profits surge last year on the back of its most successful fourth-quarter performance in years.