Up to 120 textile workers at Cavalier Specialty Yarn Co's North Carolina plant are to be laid off as the Canadian firm scales back operations amid the flood of imports and the strength of the Canadian dollar.