selftests/pstore: add pstore test scripts going with reboot

selftests/pstore: add pstore test scripts going with reboot

To test pstore in earnest, we have to cause kernel crash and check
pstore filesystem after reboot.

We add two scripts:
 - pstore_crash_test
     This script causes kernel crash and reboot. It is executed by
     'make run_pstore_crash' in selftests. It can also be used with kdump.
 - pstore_post_reboot_tests
     This script includes test cases which check pstore's behavior after
     crash and reboot. It is executed together with pstore_tests by
     'make run_tests [-C pstore]' in selftests.

The test cases in pstore_post_reboot_tests are currently following.

- Check pstore backend is registered
- Mount pstore filesystem
- Check dmesg/console/pmsg files exist in pstore filesystem
- Check dmesg/console files contain oops end marker
- Check pmsg file properly keeps the content written before crash
- Remove all files in pstore filesystem

Example usage is following.

  (before reboot)
  # cd /path/to/selftests
  # make run_tests -C pstore
  === Pstore unit tests (pstore_tests) ===
  UUID=b49b02cf-b0c2-4309-be43-b08c3971e37f
  ...
  selftests: pstore_tests [PASS]
  === Pstore unit tests (pstore_post_reboot_tests) ===
  UUID=953eb1bc-8e03-48d7-b27a-6552b24c5b7e
  Checking pstore backend is registered ... ok
          backend=ramoops
          cmdline=console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait mem=768M ramoops.mem_address=0x30000000 ramoops.mem_size=0x10000
  pstore_crash_test has not been executed yet. we skip further tests.
  selftests: pstore_post_reboot_tests [PASS]

  # make run_pstore_crash
  === Pstore unit tests (pstore_crash_test) ===
  UUID=93c8972d-1466-430b-8c4a-28d8681e74c6
  Checking pstore backend is registered ... ok
          backend=ramoops
          cmdline=console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait mem=768M ramoops.mem_address=0x30000000 ramoops.mem_size=0x10000
  Causing kernel crash ...
  (kernel crash and reboot)
  ...

  (after reboot)
  # make run_tests -C pstore
  === Pstore unit tests (pstore_tests) ===
  UUID=8e511e77-2285-499f-8bc0-900d9af1fbcc
  ...
  selftests: pstore_tests [PASS]
  === Pstore unit tests (pstore_post_reboot_tests) ===
  UUID=2dcc2132-4f3c-45aa-a38f-3b54bff8cef1
  Checking pstore backend is registered ... ok
          backend=ramoops
          cmdline=console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait mem=768M ramoops.mem_address=0x30000000 ramoops.mem_size=0x10000
  Mounting pstore filesystem ... ok
  Checking dmesg files exist in pstore filesystem ... ok
          dmesg-ramoops-0
          dmesg-ramoops-1
  Checking console files exist in pstore filesystem ... ok
          console-ramoops-0
  Checking pmsg files exist in pstore filesystem ... ok
          pmsg-ramoops-0
  Checking dmesg files contain oops end marker
          dmesg-ramoops-0 ... ok
          dmesg-ramoops-1 ... ok
  Checking console file contains oops end marker ... ok
  Checking pmsg file properly keeps the content written before crash ... ok
  Removing all files in pstore filesystem
          console-ramoops-0 ... ok
          dmesg-ramoops-0 ... ok
          dmesg-ramoops-1 ... ok
          pmsg-ramoops-0 ... ok
  selftests: pstore_post_reboot_tests [PASS]

Signed-off-by: Hiraku Toyooka <hiraku.toyooka.gu@hitachi.com>
Cc: Shuah Khan <shuahkh@osg.samsung.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Anton Vorontsov <anton@enomsg.org>
Cc: Colin Cross <ccross@android.com>
Cc: Kees Cook <keescook@chromium.org>
Cc: Mark Salyzyn <salyzyn@android.com>
Cc: Seiji Aguchi <seiji.aguchi.tr@hitachi.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-api@vger.kernel.org
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>