Culture Secretary Oliver Dowden says he was being 'attacked' by the arts for prioritising sports