usb: chipidea: imx: avoid unnecessary probe defer every time