Manus Island refugees fear for safety, refuse to leave centre