Oxford Industries has seen a fall in sales and earnings per share for its second quarter, as the company took a hit from restructuring charges related to its Ben Sherman group.