[PM] Fix a pretty nasty bug where the new pass manager would invalidate