Men's wear retailer Casual Male has reported a drop in first quarter net income sfter being hurt by a higher tax rate.