ktest: Add CONFIG_BISECT_CHECK option

The config-bisect can take a bad config and bisect it down to find out
what config actually breaks the config. But as all tests will apply a
minconfig (defined by a user) to apply before booting, it is possible
that the minconfig could actually make the bad config work (minconfigs
can disable configs). The end result is that the config bisect test will
not find a config that breaks. This can be rather frustrating to the
user.

The CONFIG_BISECT_CHECK option, when set to 1, will make sure that the
bad config (with the minconfig applied) still fails before trying to
bisect.

And yes, I did get burned by this.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

diff --git a/tools/testing/ktest/ktest.pl b/tools/testing/ktest/ktest.pl
index e91702eee580aabfbe69cbb9781f1de4b726e64d..8ce58d715aef9cce3a5d2334f88515ae0d58caf2 100755 (executable)
--- a/tools/testing/ktest/ktest.pl
+++ b/tools/testing/ktest/ktest.pl
@@ -173,6 +173,7 @@ my $bisect_check;
 
 my $config_bisect;
 my $config_bisect_type;
+my $config_bisect_check;
 
 my $patchcheck_type;
 my $patchcheck_start;
@@ -283,6 +284,7 @@ my %option_map = (
 
     "CONFIG_BISECT"            => \$config_bisect,
     "CONFIG_BISECT_TYPE"       => \$config_bisect_type,
+    "CONFIG_BISECT_CHECK"      => \$config_bisect_check,
 
     "PATCHCHECK_TYPE"          => \$patchcheck_type,
     "PATCHCHECK_START"         => \$patchcheck_start,
@@ -2743,6 +2745,18 @@ sub config_bisect {
        }
     }
     my $ret;
+
+    if (defined($config_bisect_check) && $config_bisect_check) {
+       doprint " Checking to make sure bad config with min config fails\n";
+       create_config keys %config_list;
+       $ret = run_config_bisect_test $config_bisect_type;
+       if ($ret) {
+           doprint " FAILED! Bad config with min config boots fine\n";
+           return -1;
+       }
+       doprint " Bad config with min config fails as expected\n";
+    }
+
     do {
        $ret = run_config_bisect;
     } while (!$ret);

diff --git a/tools/testing/ktest/sample.conf b/tools/testing/ktest/sample.conf
index 19754e0c8c64226acc5a27b2aeffd0b15a57b6be..e4a12da75ba6a0f3737fec7ca37f12acd9238b3d 100644 (file)
--- a/tools/testing/ktest/sample.conf
+++ b/tools/testing/ktest/sample.conf
@@ -1077,6 +1077,12 @@
 #  can specify it with CONFIG_BISECT_GOOD. Otherwise
 #  the MIN_CONFIG is the base.
 #
+# CONFIG_BISECT_CHECK (optional)
+#  Set this to 1 if you want to confirm that the config ktest
+#  generates (the bad config with the min config) is still bad.
+#  It may be that the min config fixes what broke the bad config
+#  and the test will not return a result.
+#
 # Example:
 #   TEST_START
 #   TEST_TYPE = config_bisect