Impairment and restructuring charges have pushed women's wear retailer New York & Company Inc to a fourth quarter loss.