Labour: Palace race row was 'casual racism'