Apparel group Oxford Industries Inc has agreed to acquire London-based fashion designer Ben Sherman Ltd for about $146 million in cash.