Men's underwear and pants are slimming down, but not in terms of dollar growth, according to a new survey.