Perrottet and Albanese met with ‘a bit of frustration’ from flood-hit locals