People have 'moved on' from Brexit, Health Secretary says