Men's apparel retailer The Men's Wearhouse has ousted founder George Zimmer from his position as executive chairman.