Up to 70 clothing workers are set to lose their jobs with the closure of an apparel factory in Victoria, worried union chiefs said on Thursday.