Former House Speaker Pelosi was one of the first political figures to tell Swalwell to drop out, saying the handling of accusations was "best done outside of a gubernatorial campaign.”