Clothing maker Hartmarx reported a dive in fourth-quarter profit after shutting a plant and liquidating discontinued brands.