Destination XL Group has moved to a net loss in its fourth quarter as a result of one-off charges and a sluggish retail environment.