Canadian fashion retailer Le Château has reported a drop in annual profit, following weaker sales during the year.