sched, x86: Provide a per-cpu preempt_count implementation
authorPeter Zijlstra <peterz@infradead.org>
Wed, 14 Aug 2013 12:51:00 +0000 (14:51 +0200)
committerIngo Molnar <mingo@kernel.org>
Wed, 25 Sep 2013 12:07:57 +0000 (14:07 +0200)
commitc2daa3bed53a81171cf8c1a36db798e82b91afe8
treea7a7b1676e3101b986276f9ec0eab0ab5183e1ed
parenta233f1120c37724938f7201fe2353b2577adaaf9
sched, x86: Provide a per-cpu preempt_count implementation

Convert x86 to use a per-cpu preemption count. The reason for doing so
is that accessing per-cpu variables is a lot cheaper than accessing
thread_info variables.

We still need to save/restore the actual preemption count due to
PREEMPT_ACTIVE so we place the per-cpu __preempt_count variable in the
same cache-line as the other hot __switch_to() variables such as
current_task.

NOTE: this save/restore is required even for !PREEMPT kernels as
cond_resched() also relies on preempt_count's PREEMPT_ACTIVE to ignore
task_struct::state.

Also rename thread_info::preempt_count to ensure nobody is
'accidentally' still poking at it.

Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/n/tip-gzn5rfsf8trgjoqx8hyayy3q@git.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
arch/x86/include/asm/Kbuild
arch/x86/include/asm/preempt.h [new file with mode: 0644]
arch/x86/include/asm/thread_info.h
arch/x86/kernel/asm-offsets.c
arch/x86/kernel/cpu/common.c
arch/x86/kernel/entry_32.S
arch/x86/kernel/entry_64.S
arch/x86/kernel/irq_32.c
arch/x86/kernel/process_32.c
arch/x86/kernel/process_64.c