netdev-vport: Actually destroy the vport when "destroy" is called