Fix a bug where we were counting the alias sets as completely used