rcu: Adjust ->lock acquisition for tasks no longer migrating

Tasks are no longer migrated away from a given rcu_node structure
when all CPUs corresponding to that rcu_node structure have gone offline.
This means that rcu_read_unlock_special() no longer needs to loop
retrying rcu_node ->lock acquisition because the current task is
guaranteed to stay put.

This commit takes a small and paranoid step towards relying on this
guarantee by placing a WARN_ON_ONCE() just after the early exit from
the lock-acquisition loop.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 58b1ebdc43879f0b5bd04d833f949e8d15d0a0e8..c8340e929eb46ed9920650d7eb3801f482628d9f 100644 (file)
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -307,9 +307,11 @@ void rcu_read_unlock_special(struct task_struct *t)
                t->rcu_read_unlock_special.b.blocked = false;
 
                /*
-                * Remove this task from the list it blocked on.  The
-                * task can migrate while we acquire the lock, but at
-                * most one time.  So at most two passes through loop.
+                * Remove this task from the list it blocked on.  The task
+                * now remains queued on the rcu_node corresponding to
+                * the CPU it first blocked on, so the first attempt to
+                * acquire the task's rcu_node's ->lock will succeed.
+                * Keep the loop and add a WARN_ON() out of sheer paranoia.
                 */
                for (;;) {
                        rnp = t->rcu_blocked_node;
@@ -317,6 +319,7 @@ void rcu_read_unlock_special(struct task_struct *t)
                        smp_mb__after_unlock_lock();
                        if (rnp == t->rcu_blocked_node)
                                break;
+                       WARN_ON_ONCE(1);
                        raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
                }
                empty_norm = !rcu_preempt_blocked_readers_cgp(rnp);