clk: Only recalculate the rate if needed