Ex-inspector general: Blowback came from Clinton allies