Robust Control with Perception in the Loop: Towards Open-World Manipulation