Big and tall men's wear retailer Casual Male has reported strong profit growth for the fourth quarter, saying customer visits had recovered since the downturn.