Men's apparel retailer Casual Male Retail Group Inc on Thursday announced it swung to a second quarter profit thanks to tight cost control and higher sales.