usb: dwc2/gadget: disable clock when it's not needed