spi: spidev: Restore all SPI mode flags on ioctl failure