The worst time to hold semi stocks is when numbers start to go down, says Institutional Investor's t