Restrictions on the prime to p fundamental group of a smooth projective variety