spi: dw: remove a NULL check when call ->remove()