The proposed merger between apparel company Hampshire Group and investor NAF Holdings II has been called off at the last minute - with each side blaming the other.