arm64: Add APM X-Gene SoC 15Gbps Multi-purpose PHY DTS entries

This patch adds the DTS entries for the APM X-Gene SoC 15Gbps Multi-purpose
PHY driver. The PHY for SATA controller 2 and 3 are enabled by default.

Signed-off-by: Loc Ho <lho@apm.com>
Signed-off-by: Tuan Phan <tphan@apm.com>
Signed-off-by: Suman Tripathi <stripathi@apm.com>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Tejun Heo <tj@kernel.org>
(cherry picked from commit 71b70ee9350f239ea021bbb737771ebd5d02c020)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: rwsem: use asm-generic rwsem implementation

asm-generic offers an atomic-add based rwsem implementation, which
can avoid the need for heavier, spinlock-based synchronisation on the
fast path.

This patch makes use of the optimised implementation for arm64 CPUs.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit c209f79940ac0c75ae8d2f503a2b9d86255e266c)
Signed-off-by: Mark Brown <broonie@linaro.org>

asm-generic: rwsem: de-PPCify rwsem.h

asm-generic/rwsem.h used to live under arch/powerpc. During its
liberation to common code, a few references to its former home where
preserved, in particular the definition of RWSEM_ACTIVE_MASK is
predicated on CONFIG_PPC64.

This patch updates the ifdefs and comments to architecturally neutral
versions.

Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Richard Kuo <rkuo@codeaurora.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit e172800e5d3162f97d332b3745e3743ce150ec48)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: smp: make local symbol static

Make smp_spin_table_cpu_postboot() static, because this function
is used only in this file.

Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 7184659bed3090248e382d98a49a3c1bcfe11174)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: debug: make local symbols static

Make local symbols static, because these are used only in this
file.

Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 242c04bc4be959ae28618772e439c27e87a7d880)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: barriers: add dmb barrier

Commit 8adbf57fc429 ("irqchip: gic: use dmb ishst instead of dsb when
raising a softirq") added an explicit dmb(...) call to the GIC driver.

This patch adds a simple dmb() macro to arm64, which expands to a DMB SY
instruction.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit d152d22a18c240286c19997a6249ee76ea055926)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: remove unnecessary cache flush at boot

Currently we flush the entire dcache at boot within __cpu_setup, but
this is unnecessary as the booting protocol demands that the dcache is
invalid and off upon entering the kernel. The presence of the cache
flush only serves to hide bugs in bootloaders, and is not safe in the
presence of SMP.

In an SMP boot scenario the CPUs enter coherency outside of the kernel,
and the primary CPU enables its caches before bringing up secondary
CPUs. Therefore if any secondary CPU has an entry in its cache (in
violation of the boot protocol), the primary CPU might snoop it even if
the secondary CPU's cache is disabled. The boot-time cache flush only
serves to hide a firmware bug, and slows down a cpu boot unnecessarily.

This patch removes the unnecessary boot-time cache flush.

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>
[catalin.marinas@arm.com: make __flush_dcache_all local only]
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit bff705950e2cdcf35641dee35eb14bad9ed49e8f)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Fix !CONFIG_SMP kernel build

Commit fb4a96029c8a (arm64: kernel: fix per-cpu offset restore on
resume) uses per_cpu_offset() unconditionally during CPU wakeup,
however, this is only defined for the SMP case.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Reported-by: Dave P Martin <Dave.Martin@arm.com>
(cherry picked from commit b57fc9e80692043e2a3a74e1d2c047eb700dcd0c)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: remove return value form psci_init()

psci_init() is written to return err code if something goes wrong. However,
the single user, setup_arch(), doesn't care about it. Moreover, every error
path is supplied with a clear message which is enough for pleasant debugging.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 64b4f60f497058f1c6ba118a0260249ee5c091a6)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: remove redundant "psci:" prefixes

Since 652af899799354049b273af897b798b8f03fdd88 "arm64: factor out spin-table
boot method" psci prefix's been introduced. We have a common pr_fmt, so clean
them up.

Signed-off-by: Vladimir Murzin <vladimir.murzin@arm.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 288ac26cc2334e5e6ecad6416e9bf750691afd84)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: vdso: clean up vdso_pagelist initialization

Remove some unnecessary bits that were apparently carried over from
another architecture's implementation:

- No need to get_page() the vdso text/data - these are part of the
  kernel image.
- No need for ClearPageReserved on the vdso text.
- No need to vmap the first text page to check the ELF header - this
  can be done through &vdso_start.

Also some minor cleanup:
- Use kcalloc for vdso_pagelist array allocation.
- Don't print on allocation failure, slab/slub will do that for us.

Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 16fb1a9bec6126162560f159df449e4781560807)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Change misleading function names in dma-mapping

arm64_swiotlb_alloc/free_coherent name can be misleading
somtimes with CMA support being enabled after this
patch (c2104debc235b745265b64d610237a6833fd53)

Change this name to be more generic:
__dma_alloc/free_coherent

Signed-off-by: Ritesh Harjani <ritesh.harjani@gmail.com>
[catalin.marinas@arm.com: renamed arm64_swiotlb_dma_ops to coherent_swiotlb_dma_ops]
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit bb10eb7b4d176f408d45fb492df28bed2981a1f3)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Fix the soft_restart routine

Change the soft_restart() routine to call cpu_reset() at its identity mapped
physical address.

The cpu_reset() routine must be called at its identity mapped physical address
so that when the MMU is turned off the instruction pointer will be at the correct
location in physical memory.

Signed-off-by: Geoff Levand <geoff@infradead.org> for Huawei, Linaro
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 09024aa61e1bc994404683e2e5b363484a15dd12)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Extend the idmap to the whole kernel image

This patch changes the idmap page table creation during boot to cover
the whole kernel image, allowing functions like cpu_reset() to be safely
called with the physical address.

This patch also simplifies the create_block_map asm macro to no longer
take an idmap argument and always use the phys/virt/end parameters. For
the idmap case, phys == virt.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit ea8c2e1124457f266f82effc3e6558552527943a)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: enable processor debug state for secondary cpus

processor debug state PSTATE.D is unmasked in smp call
clear_os_lock for secondary cpus. So debug state is still
masked in normal kernel context.  With this patch, unmask
debug state on secondary boot for the cpus in normal kernel
context. Now kgdb tests passed with multicore.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit d8ed442a009ecfe155b57d58f231db3d6084633d)
Signed-off-by: Mark Brown <broonie@linaro.org>
Conflicts:
arch/arm64/kernel/smp.c

Merge remote-tracking branch 'lsk/v3.10/topic/arm64-kgdb' into lsk-v3.10-arm64-misc

Conflicts:
arch/arm64/include/asm/debug-monitors.h
arch/arm64/kernel/debug-monitors.c

arm64: Extend the PCI I/O space to 16MB

The patch moves the PCI I/O space (currently at 64K) before the
earlyprintk mapping and extends it to 16MB.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 22bd1c91fe13d59cff734b69b6757adcfbd8dee9)
Signed-off-by: Mark Brown <broonie@linaro.org>
Conflicts:
Documentation/arm64/memory.txt

arm64, jump label: detect %c support for ARM64

As commit a9468f30b5eac6 "ARM: 7333/2: jump label: detect %c
support for ARM", this patch detects the same thing for ARM64
because some ARM64 GCC versions have the same issue.

Some versions of ARM64 GCC which do support asm goto, do not
support the %c specifier. Since we need the %c to support jump
labels on ARM64, detect that too in the asm goto detection script
to avoid build errors with these versions.

Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Jiang Liu <liuj97@gmail.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit f3c003f72dfb2497056bcbb864885837a1968ed5)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: KGDB: Add KGDB config

Add HAVE_ARCH_KGDB for arm64 Kconfig

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 9529247db9ecfc5a723e17093614e7437ab0d5bd)
Signed-off-by: Mark Brown <broonie@linaro.org>
Conflicts:
arch/arm64/Kconfig

misc: debug: remove compilation warnings

typecast instruction_pointer macro to unsigned long to
resolve following compiler warnings like
warning: format '%lx' expects argument of type 'long unsigned int',
but argument 2 has type 'u64' [-Wformat]

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 58dcc204f18af2821f683b235bb376f9db2557f5)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: KGDB: Add step debugging support

Add KGDB software step debugging support for EL1 debug
in AArch64 mode.

KGDB registers step debug handler with debug monitor.
On receiving 'step' command from GDB tool, target enables
software step debugging and step address is updated in ELR.

Software Step debugging is disabled when 'continue' command
is received

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
Reviewed-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 44679a4f142b69ae0c68ed815a48bbd164827281)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: KGDB: Add Basic KGDB support

Add KGDB debug support for kernel debugging.
With this patch, basic KGDB debugging is possible.GDB register
layout is updated and GDB tool can establish connection with
target and can set/clear breakpoints.

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
Reviewed-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit bcf5763b0d58d20e288ac52f96cbd7788e262cac)
Signed-off-by: Mark Brown <broonie@linaro.org>
Conflicts:
arch/arm64/kernel/Makefile

arm64: Add macros to manage processor debug state

Add macros to enable and disable to manage PSTATE.D
for debugging. The macros local_dbg_save and local_dbg_restore
are moved to irqflags.h file

KGDB boot tests fail because of PSTATE.D is masked.
unmask it for debugging support

Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@caviumnetworks.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit c7db4ff5d2b459a579d348532a92fd5885520ce6)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: defconfig: Expand default enabled features

FPGA implementations of the Cortex-A57 and Cortex-A53 are now available
in the form of the SMM-A57 and SMM-A53 Soft Macrocell Models (SMMs) for
Versatile Express. As these attach to a Motherboard Express V2M-P1 it
would be useful to have support for some V2M-P1 peripherals enabled by
default.

Additionally a couple of of features have been introduced since the last
defconfig update (CMA, jump labels) that would be good to have enabled
by default to ensure they are build and boot tested.

This patch updates the arm64 defconfig to enable support for these
devices and features. The arm64 Kconfig is modified to select
HAVE_PATA_PLATFORM, which is required to enable support for the
CompactFlash controller on the V2M-P1.

A few options which don't need to appear in defconfig are trimmed:

* BLK_DEV - selected by default
* EXPERIMENTAL - otherwise gone from the kernel
* MII - selected by drivers which require it
* USB_SUPPORT - selected by default

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 55834a773fe343912b705bef8114ec93fd337188)
Signed-off-by: Mark Brown <broonie@linaro.org>
Conflicts:
arch/arm64/configs/defconfig

arm64: asm: remove redundant "cc" clobbers

cbnz/tbnz don't update the condition flags, so remove the "cc" clobbers
from inline asm blocks that only use these instructions to implement
conditional branches.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 95c4189689f92fba7ecf9097173404d4928c6e9b)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: barriers: allow dsb macro to take option parameter

The dsb instruction takes an option specifying both the target access
types and shareability domain.

This patch allows such an option to be passed to the dsb macro,
resulting in potentially more efficient code. Currently the option is
ignored until all callers are updated (unlike ARM, the option is
mandated by the assembler).

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 4a7ac12eedd190cdf071e61145defa73df1675c0)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: simplify pgd_alloc

Currently pgd_alloc has a redundant NULL check in its return path that
can be removed with no ill effects. With that removed it's also possible
to return early and eliminate the new_pgd temporary variable.

This patch applies said modifications, making the logic of pgd_alloc
correspond 1-1 with that of pgd_free.

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 883d50a0ed403446437444a495356ce31e1197a3)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: fix typo: s/SERRROR/SERROR/

Somehow SERROR has acquired an additional 'R' in a couple of headers.
This patch removes them before they spread further. As neither instance
is in use yet, no other sites need to be fixed up.

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit bfb67a5606376bb32cb6f93dc05cda2e8c2038a5)
Signed-off-by: Mark Brown <broonie@linaro.org>
Conflicts:
arch/arm64/include/asm/kvm_arm.h

arm64: mm: fix the function name in comment of cpu_do_switch_mm

Fix the function name of comment of cpu_do_switch_mm,
because cpu_do_switch_mm is the correct name.

Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 812944e91dbbfeadaeeb4443a5560a7f45648f0b)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: mm: fix the function name in comment of __flush_dcache_area

Fix the function name of comment of __flush_dcache_area,
because __flush_dcache_area is the correct name. Also,
the missing variable 'size' is added to the comment.

Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 03324e6e6e66ebd171d9b4b90fd6a2655980dc13)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: mm: use ubfm for dcache_line_size

Use 'ubfm' for the bitfield move instruction; thus, single
instruction can be used instead of two instructions, when
getting the minimum D-cache line size from CTR_EL0 register.

Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit bd5f6dc304a054ccdc8dab43bef5e41d9a575b61)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Remove unused __data_loc variable

The __data_loc variable is an unused left over from the 32 bit arm implementation.
Remove that variable and adjust the __mmap_switched startup routine accordingly.

Signed-off-by: Geoff Levand <geoff@infradead.org> for Huawei, Linaro
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit b22cf637bbaf99d4caf9908997a32f91cdcfae52)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Remove outdated comment

Code referenced in the comment has moved to arch/arm64/kernel/cputable.c

Signed-off-by: Liviu Dudau <Liviu.Dudau@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 81cac699440fc3707fd80f16bf34a7e506d41487)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: support single-step and breakpoint handler hooks

AArch64 Single Steping and Breakpoint debug exceptions will be
used by multiple debug framworks like kprobes & kgdb.

This patch implements the hooks for those frameworks to register
their own handlers for handling breakpoint and single step events.

Reworked the debug exception handler in entry.S: do_dbg to route
software breakpoint (BRK64) exception to do_debug_exception()

Signed-off-by: Sandeepa Prabhu <sandeepa.prabhu@linaro.org>
Signed-off-by: Deepak Saxena <dsaxena@linaro.org>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit ee6214cec7818867f368c35843ea1f3dffcbb57c)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: debug: consolidate software breakpoint handlers

The software breakpoint handlers are hooked in directly from ptrace,
which makes it difficult to add additional handlers for things like
kprobes and kgdb.

This patch moves the handling code into debug-monitors.c, where we can
dispatch to different debug subsystems more easily.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 1442b6ed249d2b3d2cfcf45b65ac64393495c96c)
Signed-off-by: Mark Brown <broonie@linaro.org>

ARM64: fix framepointer check in unwind_frame

We need at least 24 bytes above frame pointer.

Signed-off-by: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 26920dd2da79a3207803da9453c0e6c82ac968ca)
Signed-off-by: Mark Brown <broonie@linaro.org>

ARM64: check stack pointer in get_wchan

get_wchan() is lockless. Task may wakeup at any time and change its own stack,
thus each next stack frame may be overwritten and filled with random stuff.

Signed-off-by: Konstantin Khlebnikov <k.khlebnikov@samsung.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 408c3658b0d49315974ce8b5aed385c8e1527595)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: kconfig: select HAVE_EFFICIENT_UNALIGNED_ACCESS

ARMv8 CPUs can perform efficient unaligned memory accesses in hardware
and this feature is relied up on by code such as the dcache
word-at-a-time name hashing.

This patch selects HAVE_EFFICIENT_UNALIGNED_ACCESS for arm64.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 50afc33a90e710c02d9bbf2f3673936365f0e690)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: dcache: select DCACHE_WORD_ACCESS for little-endian CPUs

DCACHE_WORD_ACCESS uses the word-at-a-time API for optimised string
comparisons in the vfs layer.

This patch implements support for load_unaligned_zeropad in much the
same way as has been done for ARM, although big-endian systems are also
supported.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 7bc13fd33adb9536bd73965cd46bbf7377df097c)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: futex: ensure .fixup entries are sufficiently aligned

AArch64 instructions must be 4-byte aligned, so make sure this is true
for the futex .fixup section.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 4da7a56c59f28e27e8dcff61b5d7b05f6e203606)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: use generic strnlen_user and strncpy_from_user functions

This patch implements the word-at-a-time interface for arm64 using the
same algorithm as ARM. We use the fls64 macro, which expands to a clz
instruction via a compiler builtin. Big-endian configurations make use
of the implementation from asm-generic.

With this implemented, we can replace our byte-at-a-time strnlen_user
and strncpy_from_user functions with the optimised generic versions.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 12a0ef7b0ac38677bd2d85f33df5ca0a57868819)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: percpu: implement optimised pcpu access using tpidr_el1

This patch implements optimised percpu variable accesses using the
el1 r/w thread register (tpidr_el1) along the same lines as arch/arm/.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 7158627686f02319c50c8d9d78f75d4c8d126ff2)
Signed-off-by: Mark Brown <broonie@linaro.org>

Merge remote-tracking branch 'lsk/v3.10/topic/arm64-cpu' into lsk-v3.10-arm64-misc

Conflicts:
arch/arm64/kernel/smp.c

arm64: make default NR_CPUS 8

Rather than continue to add per platform defaults, make the default a
likely common core count. 8 is also the default for x86.

Signed-off-by: Rob Herring <rob.herring@calxeda.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 62aceb8ff4b3f442575eb7e23629da36020dca77)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: ensure completion of TLB invalidatation

Currently there is no dsb between the tlbi in __cpu_setup and the write
to SCTLR_EL1 which enables the MMU in __turn_mmu_on. This means that the
TLB invalidation is not guaranteed to have completed at the point
address translation is enabled, leading to a number of possible issues
including incorrect translations and TLB conflict faults.

This patch moves the tlbi in __cpu_setup above an existing dsb used to
synchronise I-cache invalidation, ensuring that the TLBs have been
invalidated at the point the MMU is enabled.

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 3cea71bc6b470372ae407881b87128aadf0afec0)
Signed-off-by: Mark Brown <broonie@linaro.org>

ARM64: /proc/interrupts: display IPIs of online CPUs only

The non-IPI interrupts are displayed only for the online cpus from
show_interrupts in kernel/irq/proc.c before calling arch_show_interrupts().
As a result, the column headers and the IPI count don't match if any
CPU is offline.

This patch fixes show_ipi_list to display IPIs for online CPUs only.

Signed-off-by: Sudeep KarkadaNagesha <sudeep.karkadanagesha@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 67317c2689567c24d18e0dd43ab6d409fd42dc6e)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: locks: Remove CONFIG_GENERIC_LOCKBREAK

Commit 52ea2a560a9d (arm64: locks: introduce ticket-based spinlock
implementation) introduces the arch_spin_is_contended() function making
CONFIG_GENERIC_LOCKBREAK unnecessary.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>
(cherry picked from commit 61c77e0802719efce8966619cdc4234de7f252c1)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: compat: Clear the IT state independent of the 32-bit ARM or Thumb-2 mode

The ARM architecture reference specifies that the IT state bits in the
PSR must be all zeros in ARM mode or behavior is unspecified. If an ARM
function is registered as a signal handler, and that signal is delivered
inside a block of instructions following an IT instruction, some of the
instructions at the beginning of the signal handler may be skipped if
the IT state bits of the Program Status Register are not cleared by the
kernel.

Signed-off-by: T.J. Purtell <tj@mobisocial.us>
[catalin.marinas@arm.com: code comment and commit log updated]
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit aa62c2091129af81a172350b718eb35d5448cebc)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Use 42-bit address space with 64K pages

This patch expands the VA_BITS to 42 when the 64K page configuration is
enabled allowing 2TB kernel linear mapping. Linux still uses 2 levels of
page tables in this configuration with pgd now being a full page.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
(cherry picked from commit 847264fb7e73ade5b5e4b6eea3daa243a1f5217e)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: fix access to preempt_count from assembly code

preempt_count is defined as an int. Oddly enough, we access it
as a 64bit value. Things become interesting when running a BE
kernel, and looking at the current CPU number, which is stored
as an int next to preempt_count. Like in a per-cpu interrupt
handler, for example...

Using a 32bit access fixes the issue for good.

Cc: Matthew Leach <matthew.leach@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 717321fcb58ed95169bf344ae47ac6098ba5dfbe)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: use generic RW_DATA_SECTION macro in linker script

The .data section in the arm64 linker script currently lacks a
definition for page-aligned data. This leads to a .page_aligned
section being placed between the end of data and start of bss.
This patch corrects that by using the generic RW_DATA_SECTION
macro which includes support for page-aligned data.

Signed-off-by: Mark Salter <msalter@redhat.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 3c620626c0cd4cfca856d70a846398275b48a768)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: update 32-bit kuser helpers to ARMv8

This patch updates the barrier semantics in the kuser helper functions
to take advantage of the ARMv8 additions to AArch32, which are
guaranteed to be available in situations where these functions will be
called.

Note that this slightly changes the cmpxchg functions in that they are
no longer necessarily full barriers if they return 1. However, the
documentation only states they include their own barriers "as needed",
not that they are obligated to act as a full barrier for the caller.

Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Acked-by: Will Deacon <will.deacon@arm.com>
CC: Matthew Leach <matthew.leach@arm.com>
CC: Dave Martin <dave.martin@arm.com>
CC: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit d0f38f9130b7683e39611c5a661349e301ee43c8)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Export __copy_in_user() to modules

This function may be called from loadable modules, so it needs
exporting.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Reported-by: Loc Ho <lho@apm.com>
(cherry picked from commit 2a3f912c782f2364f5e5813ab66ca6c92fb43acb)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: cmpxchg: implement cmpxchg64_relaxed

This patch introduces cmpxchg64_relaxed for arm64 using the existing
cmpxchg_local macro, which performs a cmpxchg operation (up to 64 bits)
without barrier semantics.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit cf10b79a7d88edc689479af989b3a88e9adf07ff)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: lockref: add support for lockless lockrefs using cmpxchg

Our spinlocks are only 32-bit (2x16-bit tickets) and our cmpxchg can
deal with 8-bytes (as one would hope!).

This patch wires up the cmpxchg-based lockless lockref implementation
for arm64.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 5686b06cea34e31ec0a549d9b5ac00776e8e8d6d)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: locks: introduce ticket-based spinlock implementation

This patch introduces a ticket lock implementation for arm64, along the
same lines as the implementation for arch/arm/.

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 52ea2a560a9dba57fe5fd6b4726b1089751accf2)
Signed-off-by: Mark Brown <broonie@linaro.org>

clk: arm64: Add DTS clock entry for APM X-Gene Storm SoC

clk: arm64: Add DTS clock entry for APM X-Gene Storm SoC with reference to
reference, PCP PLL, SoC PLL, and Ethernet clocks.

Signed-off-by: Loc Ho <lho@apm.com>
Signed-off-by: Kumar Sankaran <ksankaran@apm.com>
Signed-off-by: Vinayak Kale <vkale@apm.com>
Signed-off-by: Feng Kan <fkan@apm.com>
Signed-off-by: Mike Turquette <mturquette@linaro.org>
(cherry picked from commit 3eb15d84e355f4ea1acf0eaba11ca173de342107)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Widen hwcap to be 64 bit

Under arm64 elf_hwcap is a 32 bit quantity, but it is stored in
a 64 bit auxiliary ELF field and glibc reads hwcap as 64 bit.

This patch widens elf_hwcap to be 64 bit.

Signed-off-by: Steve Capper <steve.capper@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 25804e6a96681d5d2142058948e218999e4f547c)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Correctly report LR and SP for compat tasks

When a task crashes and we print debugging information, ensure that
compat tasks show the actual AArch32 LR and SP registers rather than the
AArch64 ones.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 6ca68e802612c87c31aa83d50c37ed0d88774a46)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Make do_bad_area() function static

This function is only called from arch/arm64/mm/fault.c.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 59f67e16e6b79697241c3fd030e3da300377893e)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: mm: permit use of tagged pointers at EL0

TCR.TBI0 can be used to cause hardware address translation to ignore the
top byte of userspace virtual addresses. Whilst not especially useful in
standard C programs, this can be used by JITs to `tag' pointers with
various pieces of metadata.

This patch enables this bit for AArch64 Linux, and adds a new file to
Documentation/arm64/ which describes some potential caveats when using
tagged virtual addresses.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit d50240a5f6ceaf690a77b0fccb17be51cfa151c2)
Signed-off-by: Mark Brown <broonie@linaro.org>

Move the EM_ARM and EM_AARCH64 definitions to uapi/linux/elf-em.h

Signed-off-by: Dan Aloni <alonid@stratoscale.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 909e3ee4119f87b85c6e1b8534b2287ed1ea3ca2)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: delay: don't bother reporting bogomips in /proc/cpuinfo

We always use a timer-backed delay loop for arm64, so don't bother
reporting a bogomips value which appears to confuse some people.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 326b16db9f69fd0d279be873c6c00f88c0a4aad5)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Fix mapping of memory banks not ending on a PMD_SIZE boundary

The map_mem() function limits the current memblock limit to PGDIR_SIZE
(the initial swapper_pg_dir mapping) to avoid create_mapping()
allocating memory from unmapped areas. However, if the first block is
within PGDIR_SIZE and not ending on a PMD_SIZE boundary, when 4K page
configuration is enabled, create_mapping() will try to allocate a pte
page. Such page may be returned by memblock_alloc() from the end of such
bank (or any subsequent bank within PGDIR_SIZE) which is not mapped yet.

The patch limits the current memblock limit to the aligned end of the
first bank and gradually increases it as more memory is mapped. It also
ensures that the start of the first bank is aligned to PMD_SIZE to avoid
pte page allocation for this mapping.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Reported-by: "Leizhen (ThunderTown, Euler)" <thunder.leizhen@huawei.com>
Tested-by: "Leizhen (ThunderTown, Euler)" <thunder.leizhen@huawei.com>
(cherry picked from commit e25208f77c2dad5a9f2ab3d3df61252a90b71afa)
Signed-off-by: Mark Brown <broonie@linaro.org>

Merge remote-tracking branch 'lsk/v3.10/topic/arm64-hugepages' into lsk-v3.10-arm64-misc

arm64: move elf notes into readonly segment

The current vmlinux.lds.S places the notes sections between the
end of rw data and start of bss. This means that _edata doesn't
really point to the end of data. Since notes are read-only, this
patch moves them to the read-only segment so that _edata does
point to the end of initialized rw data.

Signed-off-by: Mark Salter <msalter@redhat.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit c80b7ee8520606f77fbc8ced870c96659053269e)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Enable interrupts in the EL0 undef handler

do_undefinstr() has to be called with interrupts disabled since it may
read the instruction from the user address space which could lead to a
data abort and subsequent might_sleep() warning in do_page_fault().

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 2600e130b3c90c8d6c13229d3d3a14dcb898a87b)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Expand arm64 image header

Expand the arm64 image header to allow for co-existance with
PE/COFF header required by the EFI stub.  The PE/COFF format
requires the "MZ" header to be at offset 0, and the offset
to the PE/COFF header to be at offset 0x3c.  The image
header is expanded to allow 2 instructions at the beginning
to accommodate a benign intruction at offset 0 that includes
the "MZ" header, a magic number, and the offset to the PE/COFF
header.

Signed-off-by: Roy Franz <roy.franz@linaro.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 4370eec05a887b0cd4392cd5dc5b2713174745c0)
Signed-off-by: Mark Brown <broonie@linaro.org>

ARM64: include: asm: include "asm/types.h" in "pgtable-2level-types.h" and "pgtable-3level-types.h"

Need include "asm/types.h", just like arm has done, or can not pass
compiling, the related error:

  In file included from arch/arm64/include/asm/page.h:37:0,
                   from drivers/staging/lustre/include/linux/lnet/linux/lib-lnet.h:42,
                   from drivers/staging/lustre/include/linux/lnet/lib-lnet.h:44,
                   from drivers/staging/lustre/lnet/lnet/api-ni.c:38:
  arch/arm64/include/asm/pgtable-2level-types.h:19:1: error: unknown type name ‘u64
  arch/arm64/include/asm/pgtable-2level-types.h:20:1: error: unknown type name ‘u64’

Signed-off-by: Chen Gang <gang.chen@asianux.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 360b35a874f130305715b1b854b67dc40826fc91)
Signed-off-by: Mark Brown <broonie@linaro.org>

of: Specify initrd location using 64-bit

On some PAE architectures, the entire range of physical memory could reside
outside the 32-bit limit.  These systems need the ability to specify the
initrd location using 64-bit numbers.

This patch globally modifies the early_init_dt_setup_initrd_arch() function to
use 64-bit numbers instead of the current unsigned long.

There has been quite a bit of debate about whether to use u64 or phys_addr_t.
It was concluded to stick to u64 to be consistent with rest of the device
tree code. As summarized by Geert, "The address to load the initrd is decided
by the bootloader/user and set at that point later in time. The dtb should not
be tied to the kernel you are booting"

More details on the discussion can be found here:
https://lkml.org/lkml/2013/6/20/690
https://lkml.org/lkml/2012/9/13/544

Signed-off-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Acked-by: Rob Herring <rob.herring@calxeda.com>
Acked-by: Vineet Gupta <vgupta@synopsys.com>
Acked-by: Jean-Christophe PLAGNIOL-VILLARD <plagnioj@jcrosoft.com>
Signed-off-by: Grant Likely <grant.likely@linaro.org>
(cherry picked from commit 374d5c9964c10373ba39bbe934f4262eb87d7114)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: add '#ifdef CONFIG_COMPAT' for aarch32_break_handler()

If 'COMPAT' not defined, aarch32_break_handler() cannot pass compiling,
and it can work independent with 'COMPAT', so remove dummy definition.

The related error:

  arch/arm64/kernel/debug-monitors.c:249:5: error: redefinition of ‘aarch32_break_handler’
  In file included from arch/arm64/kernel/debug-monitors.c:29:0:
  /root/linux-next/arch/arm64/include/asm/debug-monitors.h:89:12: note: previous definition of ‘aarch32_break_handler’ was here

Signed-off-by: Chen Gang <gang.chen@asianux.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit c783c2815e13bbb0c0b99997cc240bd7e91b6bb8)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Add initial DTS for APM X-Gene Storm SOC and APM Mustang board

This patch adds initial DTS files required for APM Mustang board.

Signed-off-by: Kumar Sankaran <ksankaran@apm.com>
Signed-off-by: Loc Ho <lho@apm.com>
Signed-off-by: Feng Kan <fkan@apm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit ee877b5321c4dfee9dc9f2a12b19ddcd33149f6a)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Add defines for APM ARMv8 implementation

This patch adds defines for APM CPU implementer ID and APM CPU part numbers in asm/cputype.h

Signed-off-by: Kumar Sankaran <ksankaran@apm.com>
Signed-off-by: Loc Ho <lho@apm.com>
Signed-off-by: Feng Kan <fkan@apm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 4ad637a452d5683ca7ff9e9eb994ac4b7a517073)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Enable APM X-Gene SOC family in the defconfig

This patch enables APM X-Gene SOC family in the defconfig. It also enables 8250 serial driver needed by X-Gene SOC family.

Signed-off-by: Kumar Sankaran <ksankaran@apm.com>
Signed-off-by: Loc Ho <lho@apm.com>
Signed-off-by: Feng Kan <fkan@apm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit c1db16dc9e74addba4ef8c954702f13e457f0c7e)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Add Kconfig option for APM X-Gene SOC family

This patch adds arm64/Kconfig option for APM X-Gene SOC family.

Signed-off-by: Kumar Sankaran <ksankaran@apm.com>
Signed-off-by: Loc Ho <lho@apm.com>
Signed-off-by: Feng Kan <fkan@apm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 159428538323188158a6058956c16c199909d844)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64/Makefile: provide vdso_install target

Provide a vdso_install target in the arm64 Makefile, as other architectures
with a vdso do.

Signed-off-by: Kyle McMartin <kyle@redhat.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 3c01742a8ac93a3abf9b099758db970410427afd)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: debug: consolidate software breakpoint handlers

The software breakpoint handlers are hooked in directly from ptrace,
which makes it difficult to add additional handlers for things like
kprobes and kgdb.

This patch moves the handling code into debug-monitors.c, where we can
dispatch to different debug subsystems more easily.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 1442b6ed249d2b3d2cfcf45b65ac64393495c96c)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: extable: sort the exception table at build time

As is done for other architectures, sort the exception table at
build-time rather than during boot.

Since sortextable appears to be a standalone C program relying on the
host elf.h to provide EM_AARCH64, I've had to add a conditional check in
order to allow cross-compilation on machines that aren't running a
bleeding-edge libc-dev.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit adace89562c7a9645b8dc84f6e1ac7ba8756094e)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: device: add iommu pointer to device archdata

When using an IOMMU for device mappings, it is necessary to keep a
pointer between the device and the IOMMU to which it is attached in
order to obtain the correct IOMMU when attaching the device to a domain.

This patch adds an iommu pointer to the dev_archdata structure, in a
similar manner to other architectures (ARM, PowerPC, x86, ...).

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 73150c983ac1f9b7653cfd3823b1ad4a44aad3bf)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: pgtable: use pte_index instead of __pte_index

pte_index is a useful helper outside of arch/arm64, for things like the
ARM SMMU driver, so rename __pte_index to pte_index to be consistent
with both arch/arm/ and also the definitions of pmd_index and pgd_index.

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 9ab6d02fddc6831b166812956ff387d7112ff626)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: kernel: compiling issue, need delete read_current_timer()

Under arm64, we will calibrate the delay loop statically using a known
timer frequency, so delete read_current_timer(), or it will cause
compiling issue with allmodconfig.

The related error:
  ERROR: "read_current_timer" [lib/rbtree_test.ko] undefined!
  ERROR: "read_current_timer" [lib/interval_tree_test.ko] undefined!
  ERROR: "read_current_timer" [fs/ext4/ext4.ko] undefined!
  ERROR: "read_current_timer" [crypto/tcrypt.ko] undefined!

Signed-off-by: Chen Gang <gang.chen@asianux.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 6916b14ea140ff5c915895eefe9431888a39a84d)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: mm: don't bother invalidating the icache in switch_mm

We don't support software broadcast of cache maintenance operations, so
this flush is not required (__sync_icache_dcache will always affect all
CPUs).

Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 737c16dffc458e58ee7840556d43b874cb8e16a0)
Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: Fix build for __PAGE_NONE define

Simple typo.

Signed-off-by: Mark Brown <broonie@linaro.org>

arm64: mm: Add double logical invert to pte accessors

Page table entries on ARM64 are 64 bits, and some pte functions such as
pte_dirty return a bitwise-and of a flag with the pte value. If the
flag to be tested resides in the upper 32 bits of the pte, then we run
into the danger of the result being dropped if downcast.

For example:
gather_stats(page, md, pte_dirty(*pte), 1);
where pte_dirty(*pte) is downcast to an int.

This patch adds a double logical invert to all the pte_ accessors to
ensure predictable downcasting.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>

arm64: mm: Introduce PTE_WRITE

We have the following means for encoding writable or dirty ptes:

                                PTE_DIRTY       PTE_RDONLY
!pte_dirty && !pte_write        0               1
!pte_dirty && pte_write         0               1
pte_dirty && !pte_write         1               1
pte_dirty && pte_write          1               0

So we can't distinguish between writable clean ptes and read only
ptes. This can cause problems with ptes being incorrectly flagged as
read only when they are writable but not dirty.

This patch introduces a new software bit PTE_WRITE which allows us to
correctly identify writable ptes. PTE_RDONLY is now only clear for
valid ptes where a page is both writable and dirty.

Signed-off-by: Steve Capper <steve.capper@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Conflicts:
arch/arm64/include/asm/pgtable.h

arm64: mm: Remove PTE_BIT_FUNC macro

Expand out the pte manipulation functions. This makes our life easier
when using things like tags and cscope.

Signed-off-by: Steve Capper <steve.capper@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>

mm/hugetlb.c: call MMU notifiers when copying a hugetlb page range

When copy_hugetlb_page_range() is called to copy a range of hugetlb
mappings, the secondary MMUs are not notified if there is a protection
downgrade, which breaks COW semantics in KVM.

This patch adds the necessary MMU notifier calls.

Signed-off-by: Andreas Sandberg <andreas@sandberg.pp.se>
Acked-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Rik van Riel <riel@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

arm64: mm: Fix PMD_SECT_PROT_NONE definition

Modify the value of PMD_SECT_PROT_NONE to match that of PTE_NONE. This
should have been in commit 3676f9ef5481 (Move PTE_PROT_NONE higher up).

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Cc: <stable@vger.kernel.org> # 3.11+: 3676f9ef5481: arm64: Move PTE_PROT_NONE higher up
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>

arm64: Move PTE_PROT_NONE higher up

PTE_PROT_NONE means that a pte is present but does not have any
read/write attributes. However, setting the memory type like
pgprot_writecombine() is allowed and such bits overlap with
PTE_PROT_NONE. This causes mmap/munmap issues in drivers that change the
vma->vm_pg_prot on PROT_NONE mappings.

This patch reverts the PTE_FILE/PTE_PROT_NONE shift in commit
59911ca4325d (ARM64: mm: Move PTE_PROT_NONE bit) and moves PTE_PROT_NONE
together with the other software bits.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Tested-by: Steve Capper <steve.capper@linaro.org>
Cc: <stable@vger.kernel.org> # 3.11+

ARM64: mm: THP support.

Bring Transparent HugePage support to ARM. The size of a
transparent huge page depends on the normal page size. A
transparent huge page is always represented as a pmd.

If PAGE_SIZE is 4KB, THPs are 2MB.
If PAGE_SIZE is 64KB, THPs are 512MB.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>

ARM64: mm: Raise MAX_ORDER for 64KB pages and THP.

The buddy allocator has a default MAX_ORDER of 11, which is too
low to allocate enough memory for 512MB Transparent HugePages if
our base page size is 64KB.

This patch introduces MAX_ZONE_ORDER and sets it to 14 when 64KB
pages are used in conjuction with THP, otherwise the default value
of 11 is used.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>

ARM64: mm: HugeTLB support.

Add huge page support to ARM64, different huge page sizes are
supported depending on the size of normal pages:

PAGE_SIZE is 4KB:
   2MB - (pmds) these can be allocated at any time.
1024MB - (puds) usually allocated on bootup with the command line
         with something like: hugepagesz=1G hugepages=6

PAGE_SIZE is 64KB:
 512MB - (pmds) usually allocated on bootup via command line.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>

ARM64: mm: Move PTE_PROT_NONE bit.

Under ARM64, PTEs can be broadly categorised as follows:
   - Present and valid: Bit #0 is set. The PTE is valid and memory
     access to the region may fault.

   - Present and invalid: Bit #0 is clear and bit #1 is set.
     Represents present memory with PROT_NONE protection. The PTE
     is an invalid entry, and the user fault handler will raise a
     SIGSEGV.

   - Not present (file or swap): Bits #0 and #1 are clear.
     Memory represented has been paged out. The PTE is an invalid
     entry, and the fault handler will try and re-populate the
     memory where necessary.

Huge PTEs are block descriptors that have bit #1 clear. If we wish
to represent PROT_NONE huge PTEs we then run into a problem as
there is no way to distinguish between regular and huge PTEs if we
set bit #1.

To resolve this ambiguity this patch moves PTE_PROT_NONE from
bit #1 to bit #2 and moves PTE_FILE from bit #2 to bit #3. The
number of swap/file bits is reduced by 1 as a consequence, leaving
60 bits for file and swap entries.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>

ARM64: mm: Make PAGE_NONE pages read only and no-execute.

If we consider the following code sequence:

my_pte = pte_modify(entry, myprot);
x = pte_write(my_pte);
y = pte_exec(my_pte);

If myprot comes from a PROT_NONE page, then x and y will both be
true which is undesireable behaviour.

This patch sets the no-execute and read-only bits for PAGE_NONE
such that the code above will return false for both x and y.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>

ARM64: mm: Restore memblock limit when map_mem finished.

In paging_init the memblock limit is set to restrict any addresses
returned by early_alloc to fit within the initial direct kernel
mapping in swapper_pg_dir. This allows map_mem to allocate puds,
pmds and ptes from the initial direct kernel mapping.

The limit stays low after paging_init() though, meaning any
bootmem allocations will be from a restricted subset of memory.
Gigabyte huge pages, for instance, are normally allocated from
bootmem as their order (18) is too large for the default buddy
allocator (MAX_ORDER = 11).

This patch restores the memblock limit when map_mem has finished,
allowing gigabyte huge pages (and other objects) to be allocated
from all of bootmem.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>

mm: thp: Correct the HPAGE_PMD_ORDER check.

All Transparent Huge Pages are allocated by the buddy allocator.

A compile time check is in place that fails when the order of a
transparent huge page is too large to be allocated by the buddy
allocator. Unfortunately that compile time check passes when:
HPAGE_PMD_ORDER == MAX_ORDER
( which is incorrect as the buddy allocator can only allocate
memory of order strictly less than MAX_ORDER. )

This patch updates the compile time check to fail in the above
case.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Andrew Morton <akpm@linux-foundation.org>

x86: mm: Remove general hugetlb code from x86.

huge_pte_alloc, huge_pte_offset and follow_huge_p[mu]d have
already been copied over to mm.

This patch removes the x86 copies of these functions and activates
the general ones by enabling:
CONFIG_ARCH_WANT_GENERAL_HUGETLB

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Andrew Morton <akpm@linux-foundation.org>

mm: hugetlb: Copy general hugetlb code from x86 to mm.

The huge_pte_alloc, huge_pte_offset and follow_huge_p[mu]d
functions in x86/mm/hugetlbpage.c do not rely on any architecture
specific knowledge other than the fact that pmds and puds can be
treated as huge ptes.

To allow other architectures to use this code (and reduce the need
for code duplication), this patch copies these functions into mm,
replaces the use of pud_large with pud_huge and provides a config
flag to activate them:
CONFIG_ARCH_WANT_GENERAL_HUGETLB

If CONFIG_ARCH_WANT_HUGE_PMD_SHARE is also active then the
huge_pmd_share code will be called by huge_pte_alloc (othewise we
call pmd_alloc and skip the sharing code).

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Andrew Morton <akpm@linux-foundation.org>

x86: mm: Remove x86 version of huge_pmd_share.

The huge_pmd_share code has been copied over to mm/hugetlb.c to
make it accessible to other architectures.

Remove the x86 copy of the huge_pmd_share code and enable the
ARCH_WANT_HUGE_PMD_SHARE config flag. That way we reference the
general one.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Andrew Morton <akpm@linux-foundation.org>

mm: hugetlb: Copy huge_pmd_share from x86 to mm.

Under x86, multiple puds can be made to reference the same bank of
huge pmds provided that they represent a full PUD_SIZE of shared
huge memory that is aligned to a PUD_SIZE boundary.

The code to share pmds does not require any architecture specific
knowledge other than the fact that pmds can be indexed, thus can
be beneficial to some other architectures.

This patch copies the huge pmd sharing (and unsharing) logic from
x86/ to mm/ and introduces a new config option to activate it:
CONFIG_ARCH_WANTS_HUGE_PMD_SHARE

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Andrew Morton <akpm@linux-foundation.org>