spi: fsl-(e)spi: simplify cleanup code