Bangladeshi textile chiefs on Monday said the sector would be hit hard by the United States' decision to refuse its appeal against a $50 million penalty for breaking quota restrictions.