usb: host: lpc32xx: balance clk enable/disable on removal