Apparel maker Oxford Industries has reported a drop in first quarter profit as strong sales in its Tommy Bahama business were offset by weak sales in its men's wear division.