nfsd4: allow removing clients not holding state