Andrew Higginson, the chairman of online, catalogue and stores retailer N Brown Group, has stepped down from his role.