X-Git-Url: http://demsky.eecs.uci.edu/git/?a=blobdiff_plain;f=docs%2FTestingGuide.html;h=c39065a2824eb9279ef0043ff9a549880c6dd92e;hb=be22683202731127a5128db8be404edcee91defd;hp=afb441511c35648134a1737bf8a55d7fd2484d92;hpb=ec86a7af5282ebd2daf2805fadf2eb0c33cd021a;p=oota-llvm.git diff --git a/docs/TestingGuide.html b/docs/TestingGuide.html index afb441511c3..c39065a2824 100644 --- a/docs/TestingGuide.html +++ b/docs/TestingGuide.html @@ -2,311 +2,1024 @@ "http://www.w3.org/TR/html4/strict.dtd"> - LLVM Test Suite Guide + LLVM Testing Infrastructure Guide
- LLVM Test Suite Guide + LLVM Testing Infrastructure Guide
    -
  1. Overview
    2. -
  2. Requirements
    3. -
  3. Quick Start
    4. -
  4. LLVM Test Suite Organization -
    5. -
  5. LLVM Test Suite Tree
    6. -
  6. QMTest Structure
    7. -
  7. Programs Structure
    8. -
  8. Running the LLVM Tests
    9. +
  9. Overview
    10. +
  10. Requirements
    11. +
  11. LLVM testing infrastructure organization + +
    12. +
  12. Quick start + +
    13. +
  13. DejaGNU structure + +
    14. +
  14. Test suite structure
    15. +
  15. Running the test suite + +
    16. +
  16. Running the nightly tester
-

Written by John T. Criswell

+

Written by John T. Criswell, Reid Spencer, and Tanya Lattner

- +
Overview
- +
-

This document is the reference manual for the LLVM test suite. It documents -the structure of the LLVM test suite, the tools needed to use it, and how to add -and run tests.

+

This document is the reference manual for the LLVM testing infrastructure. It documents +the structure of the LLVM testing infrastructure, the tools needed to use it, +and how to add and run tests.

- -
Requirements
- + +
Requirements
+
-

In order to use the LLVM test suite, you will need all of the software +

In order to use the LLVM testing infrastructure, you will need all of the software required to build LLVM, plus the following:

-
QMTest
-
The LLVM test suite uses QMTest to organize and run tests. Note: - you will need QMTest 2.0.3 to be successful. The tests do not run with - any other version.
- -
Python
-
You will need a Python interpreter that works with QMTest. Python will - need zlib and SAX support enabled.
+
DejaGNU
+
The Feature and Regressions tests are organized and run by DejaGNU.
+
Expect
+
Expect is required by DejaGNU.
+
tcl
+
Tcl is required by DejaGNU.
- -
Quick Start
- + +
LLVM testing infrastructure organization
+ + +
+ +

The LLVM testing infrastructure contains two major categories of tests: code +fragments and whole programs. Code fragments are referred to as the "DejaGNU +tests" and are in the llvm module in subversion under the +llvm/test directory. The whole programs tests are referred to as the +"Test suite" and are in the test-suite module in subversion. +

+ +
+ + +
DejaGNU tests
+ + +
+ +

Code fragments are small pieces of code that test a specific +feature of LLVM or trigger a specific bug in LLVM. They are usually +written in LLVM assembly language, but can be written in other +languages if the test targets a particular language front end (and the +appropriate --with-llvmgcc options were used +at configure time of the llvm module). These tests +are driven by the DejaGNU testing framework, which is hidden behind a +few simple makefiles.

+ +

These code fragments are not complete programs. The code generated +from them is never executed to determine correct behavior.

+ +

These code fragment tests are located in the llvm/test +directory.

+ +

Typically when a bug is found in LLVM, a regression test containing +just enough code to reproduce the problem should be written and placed +somewhere underneath this directory. In most cases, this will be a small +piece of LLVM assembly language code, often distilled from an actual +application or benchmark.

+ +
+ + +
Test suite
+ + +
+ +

The test suite contains whole programs, which are pieces of +code which can be compiled and linked into a stand-alone program that can be +executed. These programs are generally written in high level languages such as +C or C++, but sometimes they are written straight in LLVM assembly.

+ +

These programs are compiled and then executed using several different +methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation, +etc). The output of these programs is compared to ensure that LLVM is compiling +the program correctly.

+ +

In addition to compiling and executing programs, whole program tests serve as +a way of benchmarking LLVM performance, both in terms of the efficiency of the +programs generated as well as the speed with which LLVM compiles, optimizes, and +generates code.

+ +

The test-suite is located in the test-suite Subversion module.

+ +
+ + +
Quick start
+
-

The tests are located in the LLVM source tree under the directory -llvm/test. To run all of the tests in LLVM, use the Master Makefile in -that directory:

+

The tests are located in two separate Subversion modules. The + DejaGNU tests are in the main "llvm" module under the directory + llvm/test (so you get these tests for free with the main llvm tree). + The more comprehensive test suite that includes whole +programs in C and C++ is in the test-suite module. This module should +be checked out to the llvm/projects directory (don't use another name +then the default "test-suite", for then the test suite will be run every time +you run make in the main llvm directory). +When you configure the llvm module, +the test-suite directory will be automatically configured. +Alternatively, you can configure the test-suite module manually.

+ + +
DejaGNU tests
+ +

To run all of the simple tests in LLVM using DejaGNU, use the master Makefile + in the llvm/test directory:

+ +
+
+% gmake -C llvm/test
+
+
+ +

or

+ +
+
+% gmake check
+
+
+ +

To run only a subdirectory of tests in llvm/test using DejaGNU (ie. +Transforms), just set the TESTSUITE variable to the path of the +subdirectory (relative to llvm/test):

+
- % gmake -C llvm/test
+% gmake TESTSUITE=Transforms check
+
-

To run only the code fragment tests (i.e. those that do basic testing of -LLVM), run the tests organized by QMTest:

+

Note: If you are running the tests with objdir != subdir, you +must have run the complete testsuite before you can specify a +subdirectory.

+

To run only a single test, set TESTONE to its path (relative to +llvm/test) and make the check-one target:

+ +
- % gmake -C llvm/test qmtest
+% gmake TESTONE=Feature/basictest.ll check-one
+
-

To run only the tests that compile and execute whole programs, run the -Programs tests:

+

To run the tests with Valgrind (Memcheck by default), just append +VG=1 to the commands above, e.g.:

+
- % gmake -C llvm/test/Programs
+% gmake check VG=1
+
+ + +
Test suite
+ +

To run the comprehensive test suite (tests that compile and execute whole +programs), first checkout and setup the test-suite module:

+ +
+
+% cd llvm/projects
+% svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite
+% cd ..
+% ./configure --with-llvmgccdir=$LLVM_GCC_DIR
+
+
+ +

where $LLVM_GCC_DIR is the directory where +you installed llvm-gcc, not it's src or obj +dir. The --with-llvmgccdir option assumes that +the llvm-gcc-4.2 module was configured with +--program-prefix=llvm-, and therefore that the C and C++ +compiler drivers are called llvm-gcc and llvm-g++ +respectively. If this is not the case, +use --with-llvmgcc/--with-llvmgxx to specify each +executable's location.

+ +

Then, run the entire test suite by running make in the test-suite +directory:

+ +
+
+% cd projects/test-suite
+% gmake
+
- -
LLVM Test Suite Organization
- +

Usually, running the "nightly" set of tests is a good idea, and you can also +let it generate a report by running:

+
+
+% cd projects/test-suite
+% gmake TEST=nightly report report.html
+
+
+ +

Any of the above commands can also be run in a subdirectory of +projects/test-suite to run the specified test only on the programs in +that subdirectory.

+ +
+ + +
DejaGNU structure
+ +
+

The LLVM DejaGNU tests are driven by DejaGNU together with GNU Make and are + located in the llvm/test directory. + +

This directory contains a large array of small tests + that exercise various features of LLVM and to ensure that regressions do not + occur. The directory is broken into several sub-directories, each focused on + a particular area of LLVM. A few of the important ones are:

+ + + +
+ + +
Writing new DejaGNU tests
+
+

The DejaGNU structure is very simple, but does require some information to + be set. This information is gathered via configure and is written + to a file, site.exp in llvm/test. The llvm/test + Makefile does this work for you.

+ +

In order for DejaGNU to work, each directory of tests must have a + dg.exp file. DejaGNU looks for this file to determine how to run the + tests. This file is just a Tcl script and it can do anything you want, but + we've standardized it for the LLVM regression tests. If you're adding a + directory of tests, just copy dg.exp from another directory to get + running. The standard dg.exp simply loads a Tcl + library (test/lib/llvm.exp) and calls the llvm_runtests + function defined in that library with a list of file names to run. The names + are obtained by using Tcl's glob command. Any directory that contains only + directories does not need the dg.exp file.

+ +

The llvm-runtests function lookas at each file that is passed to + it and gathers any lines together that match "RUN:". This are the "RUN" lines + that specify how the test is to be run. So, each test script must contain + RUN lines if it is to do anything. If there are no RUN lines, the + llvm-runtests function will issue an error and the test will + fail.

+ +

RUN lines are specified in the comments of the test program using the + keyword RUN followed by a colon, and lastly the command (pipeline) + to execute. Together, these lines form the "script" that + llvm-runtests executes to run the test case. The syntax of the + RUN lines is similar to a shell's syntax for pipelines including I/O + redirection and variable substitution. However, even though these lines + may look like a shell script, they are not. RUN lines are interpreted + directly by the Tcl exec command. They are never executed by a + shell. Consequently the syntax differs from normal shell script syntax in a + few ways. You can specify as many RUN lines as needed.

+ +

Each RUN line is executed on its own, distinct from other lines unless + its last character is \. This continuation character causes the RUN + line to be concatenated with the next one. In this way you can build up long + pipelines of commands without making huge line lengths. The lines ending in + \ are concatenated until a RUN line that doesn't end in \ is + found. This concatenated set of RUN lines then constitutes one execution. + Tcl will substitute variables and arrange for the pipeline to be executed. If + any process in the pipeline fails, the entire line (and test case) fails too. +

-

The LLVM test suite contains two major categories of tests: code -fragments and whole programs.

+

Below is an example of legal RUN lines in a .ll file:

+
+
+; RUN: llvm-as < %s | llvm-dis > %t1
+; RUN: llvm-dis < %s.bc-13 > %t2
+; RUN: diff %t1 %t2
+
-
Code Fragments +

As with a Unix shell, the RUN: lines permit pipelines and I/O redirection + to be used. However, the usage is slightly different than for Bash. To check + what's legal, see the documentation for the + Tcl exec + command and the + tutorial. + The major differences are:

+ + +

There are some quoting rules that you must pay attention to when writing + your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any + ' or " so they will get passed to the invoked program. For example:

+ +
+
+... | grep 'find this string'
+
+

This will fail because the ' characters are passed to grep. This would + instruction grep to look for 'find in the files this and + string'. To avoid this use curly braces to tell Tcl that it should + treat everything enclosed as one value. So our example would become:

+ +
+
+... | grep {find this string}
+
+
+ +

Additionally, the characters [ and ] are treated + specially by Tcl. They tell Tcl to interpret the content as a command to + execute. Since these characters are often used in regular expressions this can + have disastrous results and cause the entire test run in a directory to fail. + For example, a common idiom is to look for some basicblock number:

+ +
+
+... | grep bb[2-8]
+
+
+ +

This, however, will cause Tcl to fail because its going to try to execute + a program named "2-8". Instead, what you want is this:

+ +
+
+... | grep {bb\[2-8\]}
+
+
+ +

Finally, if you need to pass the \ character down to a program, + then it must be doubled. This is another Tcl special character. So, suppose + you had: + +

+
+... | grep 'i32\*'
+
+
+ +

This will fail to match what you want (a pointer to i32). First, the + ' do not get stripped off. Second, the \ gets stripped off + by Tcl so what grep sees is: 'i32*'. That's not likely to match + anything. To resolve this you must use \\ and the {}, like + this:

+ +
+
+... | grep {i32\\*}
+
+
+ +

If your system includes GNU grep, make sure +that GREP_OPTIONS is not set in your environment. Otherwise, +you may get invalid results (both false positives and false +negatives).

+ +
+ + +
The FileCheck utility
+ +
-

Code fragments are small pieces of code that test a specific feature of LLVM -or trigger a specific bug in LLVM. They are usually written in LLVM assembly -language, but can be written in other languages if the test targets a particular -language front end.

+

A powerful feature of the RUN: lines is that it allows any arbitrary commands + to be executed as part of the test harness. While standard (portable) unix + tools like 'grep' work fine on run lines, as you see above, there are a lot + of caveats due to interaction with Tcl syntax, and we want to make sure the + run lines are portable to a wide range of systems. Another major problem is + that grep is not very good at checking to verify that the output of a tools + contains a series of different output in a specific order. The FileCheck + tool was designed to help with these problems.

+ +

FileCheck (whose basic command line arguments are described in the FileCheck man page is + designed to read a file to check from standard input, and the set of things + to verify from a file specified as a command line argument. A simple example + of using FileCheck from a RUN line looks like this:

+ +
+
+; RUN: llvm-as < %s | llc -march=x86-64 | FileCheck %s
+
+
+ +

This syntax says to pipe the current file ("%s") into llvm-as, pipe that into +llc, then pipe the output of llc into FileCheck. This means that FileCheck will +be verifying its standard input (the llc output) against the filename argument +specified (the original .ll file specified by "%s"). To see how this works, +lets look at the rest of the .ll file (after the RUN line):

+ +
+
+define void @sub1(i32* %p, i32 %v) {
+entry:
+; CHECK: sub1:
+; CHECK: subl
+        %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v)
+        ret void
+}
+
+define void @inc4(i64* %p) {
+entry:
+; CHECK: inc4:
+; CHECK: incq
+        %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1)
+        ret void
+}
+
+
+ +

Here you can see some "CHECK:" lines specified in comments. Now you can see +how the file is piped into llvm-as, then llc, and the machine code output is +what we are verifying. FileCheck checks the machine code output to verify that +it matches what the "CHECK:" lines specify.

-

Code fragments are not complete programs, and they are never executed to -determine correct behavior.

+

The syntax of the CHECK: lines is very simple: they are fixed strings that +must occur in order. FileCheck defaults to ignoring horizontal whitespace +differences (e.g. a space is allowed to match a tab) but otherwise, the contents +of the CHECK: line is required to match some thing in the test file exactly.

-

The tests in the Features and Regression directories contain code -fragments.

+

One nice thing about FileCheck (compared to grep) is that it allows merging +test cases together into logical groups. For example, because the test above +is checking for the "sub1:" and "inc4:" labels, it will not match unless there +is a "subl" in between those labels. If it existed somewhere else in the file, +that would not count: "grep subl" matches if subl exists anywhere in the +file.

-
Whole Programs
+ +
The FileCheck -check-prefix option
-

Whole Programs are pieces of code which can be compiled and linked into a -stand-alone program that can be executed. These programs are generally written -in high level languages such as C or C++, but sometimes they are written -straight in LLVM assembly.

- -

These programs are compiled and then executed using several different -methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation, -etc). The output of these programs is compared to ensure that LLVM is compiling -the program correctly.

+

The FileCheck -check-prefix option allows multiple test configurations to be +driven from one .ll file. This is useful in many circumstances, for example, +testing different architectural variants with llc. Here's a simple example:

-

In addition to compiling and executing programs, whole program tests serve as -a way of benchmarking LLVM performance, both in terms of the efficiency of the -programs generated as well as the speed with which LLVM compiles, optimizes, and -generates code.

+
+
+; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \
+; RUN:              | FileCheck %s -check-prefix=X32
+; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \
+; RUN:              | FileCheck %s -check-prefix=X64
+
+define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind {
+        %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1
+        ret <4 x i32> %tmp1
+; X32: pinsrd_1:
+; X32:    pinsrd $1, 4(%esp), %xmm0
+
+; X64: pinsrd_1:
+; X64:    pinsrd $1, %edi, %xmm0
+}
+
+
-

The Programs directory contains all tests which compile and benchmark whole -programs.

+

In this case, we're testing that we get the expected code generation with +both 32-bit and 64-bit code generation.

- -
LLVM Test Suite Tree
- + +
The "CHECK-NEXT:" directive
-

Each type of test in the LLVM test suite has its own directory. The major -subtrees of the test suite directory tree are as follows:

- -
-
  • QMTest -

    This directory contains the QMTest information files. Inside this - directory are QMTest administration files and the Python code that - implements the LLVM test and database classes.

    +

    CHECK-NEXT: directives reject the input unless there is exactly one newline +between it an the previous directive. A CHECK-NEXT cannot be the first +directive in a file.

    - +
    • + + +
    The "CHECK-NOT:" directive
    + +
    + +

    The CHECK-NOT: directive is used to verify that a string doesn't occur +between two matches (or the first match and the beginning of the file). For +example, to verify that a load is removed by a transformation, a test like this +can be used:

    + +
    +
    +define i8 @coerce_offset0(i32 %V, i32* %P) {
+  store i32 %V, i32* %P
+   
+  %P2 = bitcast i32* %P to i8*
+  %P3 = getelementptr i8* %P2, i32 2
+
+  %A = load i8* %P3
+  ret i8 %A
+; CHECK: @coerce_offset0
+; CHECK-NOT: load
+; CHECK: ret i8
+}
+
    +
    - -
    QMTest Structure
    - + +
    FileCheck Pattern Matching Syntax
    -

    The LLVM test suite is partially driven by QMTest and partially -driven by GNU Make. Specifically, the Features and Regression tests -are all driven by QMTest. The Programs directory is currently -driven by a set of Makefiles.

    +

    The CHECK: and CHECK-NOT: directives both take a pattern to match. For most +uses of FileCheck, fixed string matching is perfectly sufficient. For some +things, a more flexible form of matching is desired. To support this, FileCheck +allows you to specify regular expressions in matching strings, surrounded by +double braces: {{yourregex}}. Because we want to use fixed string +matching for a majority of what we do, FileCheck has been designed to support +mixing and matching fixed string matching with regular expressions. This allows +you to write things like this:

    -

    The QMTest system needs to have several pieces of information -available; these pieces of configuration information are known -collectively as the "context" in QMTest parlance. Since the context -for LLVM is relatively large, the master Makefile in llvm/test -sets it for you.

    +
    +
    +; CHECK: movhpd	{{[0-9]+}}(%esp), {{%xmm[0-7]}}
+
    +
    -

    The LLVM database class makes the subdirectories of llvm/test a -QMTest test database. For each directory that contains tests driven by -QMTest, it knows what type of test the source file is and how to run it.

    +

    In this case, any offset from the ESP register will be allowed, and any xmm +register will be allowed.

    -

    Hence, the QMTest namespace is essentially what you see in the -Feature and Regression directories, but there is some magic that -the database class performs (as described below).

    +

    Because regular expressions are enclosed with double braces, they are +visually distinct, and you don't need to use escape characters within the double +braces like you would in C. In the rare case that you want to match double +braces explicitly from the input, you can use something ugly like +{{[{][{]}} as your pattern.

    -

    The QMTest namespace is currently composed of the following tests and test -suites:

    +
    -