Diversified apparel company Hampshire Group said it suffered a US$176,000 first-quarter loss on a "very challenging" retail market.