USB: serial: remove overly defensive port tests