Recent gains in margins and shareholder returns are "only the beginning" for apparel business Perry Ellis International, senior directors have told shareholders.