The European Union has agreed to accelerate moves to lower trade tariffs on clothing imported from tsunami-hit Sri Lanka.