This document is the reference manual for the LLVM testing infrastructure. It documents the structure of the LLVM testing infrastructure, the tools needed to @@ -64,10 +65,10 @@ use it, and how to add and run tests.
In order to use the LLVM testing infrastructure, you will need all of the software required to build LLVM, as well @@ -76,10 +77,10 @@ as Python 2.4 or later.
The LLVM testing infrastructure contains two major categories of tests: regression tests and whole programs. The regression tests are contained inside @@ -89,13 +90,11 @@ referred to as the "LLVM test suite" and are in the test-suite module in subversion.
-The regression tests are small pieces of code that test a specific feature of LLVM or trigger a specific bug in LLVM. They are usually written in LLVM @@ -119,10 +118,10 @@ application or benchmark.
The test suite contains whole programs, which are pieces of code which can be compiled and linked into a stand-alone program that can be @@ -144,11 +143,10 @@ generates code.
The test suite contains tests to check quality of debugging information. The test are written in C based languages or in LLVM assembly language.
@@ -160,11 +158,13 @@ test suite for more information . This test suite is located in theThe tests are located in two separate Subversion modules. The regressions tests are in the main "llvm" module under the directory @@ -179,7 +179,8 @@ the test-suite directory will be automatically configured. Alternatively, you can configure the test-suite module manually.
- +To run all of the LLVM regression tests, use master Makefile in the llvm/test directory:
@@ -198,7 +199,7 @@ Alternatively, you can configure the test-suite module manually.If you have Clang checked out and built, +
If you have Clang checked out and built, you can run the LLVM and Clang tests simultaneously using:
or
@@ -239,10 +240,14 @@ script which is built as part of LLVM. For example, to run theFor more information on using the 'lit' tool, see 'llvm-lit --help' or the 'lit' man page.
+To run the comprehensive test suite (tests that compile and execute whole programs), first checkout and setup the test-suite module:
@@ -292,9 +297,10 @@ that subdirectory.To run debugging information tests simply checkout the tests inside clang/test directory.
@@ -310,10 +316,14 @@ clang/test directory.The LLVM regression tests are driven by 'lit' and are located in the llvm/test directory. @@ -335,12 +345,10 @@ clang/test directory.
The regression test structure is very simple, but does require some information to be set. This information is gathered via configure and is written to a file, lit.site.cfg @@ -358,8 +366,8 @@ clang/test directory.
obtained by using Tcl's glob command. Any directory that contains only directories does not need the dg.exp file. -The llvm-runtests function lookas at each file that is passed to - it and gathers any lines together that match "RUN:". This are the "RUN" lines +
The llvm-runtests function looks at each file that is passed to + it and gathers any lines together that match "RUN:". These are the "RUN" lines that specify how the test is to be run. So, each test script must contain RUN lines if it is to do anything. If there are no RUN lines, the llvm-runtests function will issue an error and the test will @@ -376,6 +384,11 @@ clang/test directory.
shell. Consequently the syntax differs from normal shell script syntax in a few ways. You can specify as many RUN lines as needed. +lit performs substitution on each RUN line to replace LLVM tool + names with the full paths to the executable built for each tool (in + $(LLVM_OBJ_ROOT)/$(BuildMode)/bin). This ensures that lit does not + invoke any stray LLVM tools in the user's path during testing.
+Each RUN line is executed on its own, distinct from other lines unless its last character is \. This continuation character causes the RUN line to be concatenated with the next one. In this way you can build up long @@ -487,10 +500,10 @@ negatives).
A powerful feature of the RUN: lines is that it allows any arbitrary commands to be executed as part of the test harness. While standard (portable) unix @@ -556,13 +569,12 @@ is a "subl" in between those labels. If it existed somewhere else in the file, that would not count: "grep subl" matches if subl exists anywhere in the file.
-The FileCheck -check-prefix option allows multiple test configurations to be driven from one .ll file. This is useful in many circumstances, for example, @@ -593,13 +605,14 @@ both 32-bit and 64-bit code generation.
Sometimes you want to match lines and would like to verify that matches -happen on exactly consequtive lines with no other lines in between them. In +happen on exactly consecutive lines with no other lines in between them. In this case, you can use CHECK: and CHECK-NEXT: directives to specify this. If you specified a custom check prefix, just use "<PREFIX>-NEXT:". For example, something like this works as you'd expect:
@@ -633,10 +646,11 @@ directive in a file.The CHECK-NOT: directive is used to verify that a string doesn't occur between two matches (or the first match and the beginning of the file). For @@ -663,10 +677,11 @@ define i8 @coerce_offset0(i32 %V, i32* %P) {
The CHECK: and CHECK-NOT: directives both take a pattern to match. For most uses of FileCheck, fixed string matching is perfectly sufficient. For some @@ -695,10 +710,11 @@ braces explicitly from the input, you can use something ugly like
It is often useful to match a pattern and then verify that it occurs again later in the file. For codegen tests, this can be useful to allow any register, @@ -733,11 +749,12 @@ define two separate CHECK lines that match on the same line.
With a RUN line there are a number of substitutions that are permitted. In general, any Tcl variable that is available in the substitute function (in test/lib/llvm.exp) can be substituted into a RUN line. @@ -782,37 +799,10 @@ substitutions
To make RUN line writing easier, there are several shell scripts located in the llvm/test/Scripts directory. This directory is in the PATH when running tests, so you can just call these scripts using their name. For @@ -887,12 +877,13 @@ substitutions
The test-suite module contains a number of programs that can be compiled with LLVM and executed. These programs are compiled using the native compiler @@ -957,10 +948,10 @@ will help you separate benign warnings from actual test failures.
First, all tests are executed within the LLVM object directory tree. They are not executed inside of the LLVM source tree. This is because the @@ -1015,14 +1006,13 @@ test suite creates temporary files during execution.
have the suite checked out and configured, you don't need to do it again (unless the test code or configure script changes). -In order to run the External tests in the test-suite module, you must specify --with-externals. This must be done during the re-configuration step (see above), @@ -1050,10 +1040,11 @@ the test code or configure script changes).
In addition to the regular "whole program" tests, the test-suite module also provides a mechanism for compiling the programs in different ways. If the variable TEST is defined on the gmake command line, the test system will @@ -1073,10 +1064,11 @@ LLVM.
There are a number of ways to run the tests and generate output. The most simple one is simply running gmake with no arguments. This will compile and run all programs in the tree using a number of different methods @@ -1104,11 +1096,12 @@ LLVM.
Assuming you can run the test suite, (e.g. "gmake TEST=nightly report" should work), it is really easy to run optimizations or code generator @@ -1174,6 +1167,8 @@ example reports that can do fancy stuff.