drm/amdgpu: move VM manager clean into the VM code again