serial_core: Get a reference for port->tty in uart_remove_one_port()