US men's apparel and footwear business Jos A Bank Clothiers has said it remains cautious heading into the crucial Father's Day period after suffering a gross margin decline in May and swinging to a loss in its first-quarter.