serial: bfin_uart: Don't switch baud rate untill the transfer buffer is empty.