Oxford Industries saw fourth quarter profit slip as strong performances from the Lilly Pulitzer and Tommy Bahama brands failed to offset weakness at Ben Sherman.