Upscale department store retailer Neiman Marcus Group has swung to a loss in its fourth-quarter due to an increase in interest expenses and flat margins.