Refugees refuse to go to asylum seekers' camp