spi/pl022: disable port when unused