Don't crash if the load/store pointer is not a GEP.