vswitchd: Adding and removing mgmt interface breaks connection