This document is the reference manual for the LLVM test suite. It documents +the structure of the LLVM test suite, the tools needed to use it, and how to add +and run tests.
+ +In order to use the LLVM test suite, you will need all of the software +required to build LLVM, plus the following:
+ +The tests are located in two separate CVS modules. The basic feature and +regression tests are in the main "llvm" module under the directory +llvm/test. A more comprehensive test suite that includes whole +programs in C and C++ is in the llvm-test module. This module should +be checked out to the llvm/projects directory. When you +configure the llvm module, the llvm-test module +will be automatically configured. Alternatively, you can configure the llvm-test module manually.
+To run all of the simple tests in LLVM using DejaGNU, use the master Makefile in the +llvm/test directory:
++% gmake -C llvm/test ++or
+% gmake check ++
To run only a subdirectory of tests in llvm/test using DejaGNU (ie. Regression/Transforms). Just substitute the path to the subdirectory:
++% gmake -C llvm/test TESTSUITE=Regression/Transforms ++
To run the comprehensive test suite (tests that compile and execute whole +programs), run the llvm-test tests:
+ ++% cd llvm/projects +% cvs co llvm-test +% cd llvm-test +% ./configure --with-llvmsrc=$LLVM_SRC_ROOT --with-llvmobj=$LLVM_OBJ_ROOT +% gmake ++ +
The LLVM test suite contains two major categories of tests: code +fragments and whole programs. Code fragments are in the llvm module +under the directory under the llvm/test directory. The whole programs +test suite are in the llvm-test module under the main directory.
+ +Code fragments are small pieces of code that test a specific feature of LLVM +or trigger a specific bug in LLVM. They are usually written in LLVM assembly +language, but can be written in other languages if the test targets a particular +language front end.
+ +Code fragments are not complete programs, and they are never executed to +determine correct behavior.
+ +These code fragment tests are located in the llvm/test/Features and +llvm/test/Regression directories.
+ +Whole Programs are pieces of code which can be compiled and linked into a +stand-alone program that can be executed. These programs are generally written +in high level languages such as C or C++, but sometimes they are written +straight in LLVM assembly.
+ +These programs are compiled and then executed using several different +methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation, +etc). The output of these programs is compared to ensure that LLVM is compiling +the program correctly.
+ +In addition to compiling and executing programs, whole program tests serve as +a way of benchmarking LLVM performance, both in terms of the efficiency of the +programs generated as well as the speed with which LLVM compiles, optimizes, and +generates code.
+ +All "whole program" tests are located in the llvm-test CVS +module.
+ +Each type of test in the LLVM test suite has its own directory. The major +subtrees of the test suite directory tree are as follows:
+ +This directory contains sample codes that test various features of the +LLVM language. These pieces of sample code are run through various +assembler, disassembler, and optimizer passes.
+This directory contains regression tests for LLVM. When a bug is found +in LLVM, a regression test containing just enough code to reproduce the +problem should be written and placed somewhere underneath this directory. +In most cases, this will be a small piece of LLVM assembly language code, +often distilled from an actual application or benchmark.
+The llvm-test CVS module contains programs that can be compiled +with LLVM and executed. These programs are compiled using the native compiler +and various LLVM backends. The output from the program compiled with the +native compiler is assumed correct; the results from the other programs are +compared to the native program output and pass if they match.
+ +In addition for testing correctness, the llvm-test directory also +performs timing tests of various LLVM optimizations. It also records +compilation times for the compilers and the JIT. This information can be +used to compare the effectiveness of LLVM's optimizations and code +generation.
The SingleSource directory contains test programs that are only a single +source file in size. These are usually small benchmark programs or small +programs that calculate a particular value. Several such programs are grouped +together in each directory.
The MultiSource directory contains subdirectories which contain entire +programs with multiple source files. Large benchmarks and whole applications +go here.
The External directory contains Makefiles for building code that is external +to (i.e., not distributed with) LLVM. The most prominent members of this +directory are the SPEC 95 and SPEC 2000 benchmark suites. The presence and +location of these external programs is configured by the llvm-test +configure script.
Written by John T. Criswell
+ +The LLVM test suite is partially driven by DejaGNU and partially +driven by GNU Make. Specifically, the Features and Regression tests +are all driven by DejaGNU. The llvm-test +module is currently driven by a set of Makefiles.
+ +The DejaGNU structure is very simple, but does require some +information to be set. This information is gathered via configure and +is written to a file, site.exp in llvm/test. The +llvm/test +Makefile does this work for you.
+ +In order for DejaGNU to work, each directory of tests must have a +dg.exp file. This file is a program written in tcl that calls +the llvm-runtests procedure on each test file. The +llvm-runtests procedure is defined in +llvm/test/lib/llvm-dg.exp. Any directory that contains only +directories does not need the dg.exp file.
+ +In order for a test to be run, it must contain information within +the test file on how to run the test. These are called RUN +lines. Run lines are specified in the comments of the test program +using the keyword RUN followed by a colon, and lastly the +commands to execute. These commands will be executed in a bash script, +so any bash syntax is acceptable. You can specify as many RUN lines as +necessary. Each RUN line translates to one line in the resulting bash +script. Below is an example of legal RUN lines in a .ll +file:
++; RUN: llvm-as < %s | llvm-dis > %t1 +; RUN: llvm-dis < %s.bc-13 > %t2 +; RUN: diff %t1 %t2 ++
There are a couple patterns within a RUN line that the +llvm-runtest procedure looks for and replaces with the appropriate +syntax:
+There are also several scripts in the llvm/test/Scripts directory +that you might find useful when writing RUN lines.
+ +Lastly, you can easily mark a test that is expected to fail on a +specific platform by using the XFAIL keyword. Xfail lines are +specified in the comments of the test program using XFAIL, +followed by a colon, and one or more regular expressions (separated by +a comma) that will match against the target triplet for the +machine. You can use * to match all targets. Here is an example of an +XFAIL line:
++; XFAIL: darwin,sun ++ +
As mentioned previously, the llvm-test module provides three types +of tests: MultiSource, SingleSource, and External. Each tree is then subdivided +into several categories, including applications, benchmarks, regression tests, +code that is strange grammatically, etc. These organizations should be +relatively self explanatory.
+ +In addition to the regular "whole program" tests, the llvm-test +module also provides a mechanism for compiling the programs in different ways. +If the variable TEST is defined on the gmake command line, the test system will +include a Makefile named TEST.<value of TEST variable>.Makefile. +This Makefile can modify build rules to yield different results.
+ +For example, the LLVM nightly tester uses TEST.nightly.Makefile to +create the nightly test reports. To run the nightly tests, run gmake +TEST=nightly.
+ +There are several TEST Makefiles available in the tree. Some of them are +designed for internal LLVM research and will not work outside of the LLVM +research group. They may still be valuable, however, as a guide to writing your +own TEST Makefile for any optimization or analysis passes that you develop with +LLVM.
+ +Note, when configuring the llvm-test module, you might want to +specify the following configuration options:
++
+
First, all tests are executed within the LLVM object directory tree. They +are not executed inside of the LLVM source tree. This is because the +test suite creates temporary files during execution.
+ +The master Makefile in llvm/test is capable of running only the DejaGNU driven +tests. By default, it will run all of these tests.
+ +To run only the DejaGNU driven tests, run gmake at the +command line in llvm/tests. To run a specific directory of tests, use the +TESTSUITE variable. +
+ +For example, to run the Regression tests, type +gmake TESTSUITE=Regression in llvm/tests.
+ +Note that there are no Makefiles in llvm/test/Features and +llvm/test/Regression. You must use DejaGNU from the llvm/test +directory to run them.
+ +To run the llvm-test suite, you need to use the following steps: +
+Note that the second and third steps only need to be done once. After you +have the suite checked out and configured, you don't need to do it again (unless +the test code or configure script changes).
+ +To make a specialized test (use one of the
+llvm-test/TEST.<type>.Makefiles), just run:
+gmake TEST=<type> test
For example, you could run the
+nightly tester tests using the following commands:
+ % cd llvm/projects/llvm-test + % gmake TEST=nightly test ++ +
Regardless of which test you're running, the results are printed on standard +output and standard error. You can redirect these results to a file if you +choose.
+ +Some tests are known to fail. Some are bugs that we have not fixed yet; +others are features that we haven't added yet (or may never add). In DejaGNU, +the result for such tests will be XFAIL (eXpected FAILure). In this way, you +can tell the difference between an expected and unexpected failure.
+ +The tests in llvm-test have no such feature as of this time. If the +test passes, only warnings and other miscellaneous output will be generated. If +a test fails, a large <program> FAILED message will be displayed. This +will help you separate benign warnings from actual test failures.
+ +- This document is the reference manual for the LLVM test suite. It - documents the structure of the LLVM test suite, the tools needed to - use it, and how to add and run tests. -
-- In order to use the LLVM test suite, you will need all of the software - required to build LLVM, plus the following: -
- -- The tests are located in the LLVM source tree under the directory - llvm/test. To run all of the tests in LLVM, use the Master - Makefile in that directory: -
-- % make -C llvm/test -- -
- To run only the code fragment tests (i.e. those that do basic testing of - LLVM), run the tests organized by QMTest: -
- -- % make -C llvm/test qmtest -- -
- To run only the tests that compile and execute whole programs, run the - Programs tests: -
- -- % make -C llvm/test/Programs --
The LLVM test suite contains two major categories of tests: code - fragments and whole programs.
-- Code fragments are small pieces of code that test a specific - feature of LLVM or trigger a specific bug in LLVM. They are - usually written in LLVM assembly language, but can be - written in other languages if the test targets a - particular language front end. -
- Code fragments are not complete programs, and they are - never executed to determine correct behavior. -
- The tests in the Features and - Regression directories contain code fragments. -
-- Whole Programs are pieces of code which can be compiled and - linked into a stand-alone program that can be executed. These - programs are generally written in high level languages such as C - or C++, but sometimes they are written straight in LLVM - assembly. -
- These programs are compiled and then executed using several - different methods (native compiler, LLVM C backend, LLVM JIT, - LLVM native code generation, etc). The output of these programs - is compared to ensure that LLVM is compiling the program - correctly. -
- In addition to compiling and executing programs, whole program - tests serve as a way of benchmarking LLVM performance, both in - terms of the efficiency of the programs generated as well as the - speed with which LLVM compiles, optimizes, and generates code. -
- The Programs directory contains all tests which compile and - benchmark whole programs. -
-Each type of test in the LLVM test suite has its own directory. The - major subtrees of the test suite directory tree are as follows:
- -- This directory contains sample codes that test various features - of the LLVM language. These pieces of sample code are run - through various assembler, disassembler, and optimizer passes. -
- -- This directory contains regression tests for LLVM. When a bug - is found in LLVM, a regression test containing just enough - code to reproduce the problem should be written and placed - somewhere underneath this directory. In most cases, this - will be a small piece of LLVM assembly language code, often - distilled from an actual application or benchmark. -
- -- The Programs directory contains programs that can be compiled - with LLVM and executed. These programs are compiled using the - native compiler and various LLVM backends. The output from the - program compiled with the native compiler is assumed correct; - the results from the other programs are compared to the native - program output and pass if they match. -
- In addition for testing correctness, the Programs directory - also performs timing tests of various LLVM optimizations. - It also records compilation times for the compilers and the - JIT. This information can be used to compare the - effectiveness of LLVM's optimizations and code generation. -
- The Programs directory is subdivided into several smaller - subdirectories: -
- -- The SingleSource directory contains test programs that - are only a single source file in size. These are - usually small benchmark programs or small programs that - calculate a particular value. Several such programs are - grouped together in each directory. -
- -- The MultiSource directory contains subdirectories which - contain entire programs with multiple source files. - Large benchmarks and whole applications go here. -
- -- The External directory contains Makefiles for building - code that is external to (i.e. not distributed with) - LLVM. The most prominent member of this directory is - the SPEC 2000 benchmark suite. The presence and - location of these external programs is configured by the - LLVM configure script. -
-- -
- This directory contains the QMTest information files. Inside - this directory are QMTest administration files and the Python - code that implements the LLVM test and database classes. -
-- The LLVM test suite is partially driven by QMTest and partially - driven by GNU Make. Specifically, the Features and Regression tests - are all driven by QMTest. The Programs directory is currently - driven by a set of Makefiles. -
- The QMTest system needs to have several pieces of information - available; these pieces of configuration information are known - collectively as the "context" in QMTest parlance. Since the context - for LLVM is relatively large, the master Makefile in llvm/test - sets it for you. -
- The LLVM database class makes the subdirectories of llvm/test a - QMTest test database. For each directory that contains tests driven by - QMTest, it knows what type of test the source file is and how to run it. -
- Hence, the QMTest namespace is essentially what you see in the - Feature and Regression directories, but there is some magic that - the database class performs (as described below). -
- The QMTest namespace is currently composed of the following tests and - test suites: -
- -- These are the feature tests found in the Feature directory. - They are broken up into the following categories: -
-- Assembler/Disassembler tests. These tests verify that a - piece of LLVM assembly language can be assembled into - bytecode and then disassembled into the original - assembly language code. It does this several times to - ensure that assembled output can be disassembled and - disassembler output can be assembled. It also verifies - that the give assembly language file can be assembled - correctly. -
- -- Optimizer tests. These tests verify that two of the - optimizer passes completely optimize a program (i.e. - after a single pass, they cannot optimize a program - any further). -
- -- Machine code tests. These tests verify that the LLVM - assembly language file can be translated into native - assembly code. -
- -- C code tests. These tests verify that the specified - LLVM assembly code can be converted into C source code - using the C backend. -
-- The LLVM database class looks at every file in the Feature - directory and creates a fake test hierarchy containing - Feature.<testtype>.<testname>. So, if you - add an LLVM assembly language file to the Feature directory, it - actually creates 5 new tests: assembler/disassembler, assembler, - optimizer, machine code, and C code. -
- -- These are the regression tests. There is one suite for each - subdirectory of the Regression directory. If you add a new - subdirectory there, you will need to modify, at least, the - RegressionMap variable in QMTest/llvmdb.py so - that QMTest knows how to run the tests in the new subdirectory. -
-- As mentioned previously, the Programs tree in llvm/test provides three - types of tests: MultiSource, SingleSource, and External. Each tree is - then subdivided into several categories, including applications, - benchmarks, regression tests, code that is strange grammatically, etc. - These organizations should be relatively self explanatory. -
- In addition to the regular Programs tests, the Programs tree also - provides a mechanism for compiling the programs in different ways. If - the variable TEST is defined on the gmake command line, the test system - will include a Makefile named TEST.<value of TEST - variable>.Makefile. This Makefile can modify build rules to - yield different results. -
- For example, the LLVM nightly tester uses TEST.nightly.Makefile - to create the nightly test reports. To run the nightly tests, run - gmake TEST=nightly. -
- There are several TEST Makefiles available in the tree. Some of them - are designed for internal LLVM research and will not work outside of the - LLVM research group. They may still be valuable, however, as a guide to - writing your own TEST Makefile for any optimization or analysis passes - that you develop with LLVM. -
-- First, all tests are executed within the LLVM object directory tree. - They are not executed inside of the LLVM source tree. This is - because the test suite creates temporary files during execution. -
- The master Makefile in llvm/test is capable of running both the - QMTest driven tests and the Programs tests. By default, it will run - all of the tests. -
- To run only the QMTest driven tests, run make qmtest at the - command line in llvm/tests. To run a specific qmtest, suffix the test - name with ".t" when running make. -
- For example, to run the Regression.LLC tests, type - make Regression.LLC.t in llvm/tests. -
- Note that the Makefiles in llvm/test/Features and llvm/test/Regression - are gone. You must now use QMTest from the llvm/test directory to run - them. -
- To run the Programs test, cd into the llvm/test/Programs directory and - type make. Alternatively, you can type make - TEST=<type> test to run one of the specialized tests in - llvm/test/Programs/TEST.<type>.Makefile. For example, you could - run the nightly tester tests using the following commands: -
- -- % cd llvm/test/Programs - % make TEST=nightly test -- -
- Regardless of which test you're running, the results are printed on - standard output and standard error. You can redirect these results to a - file if you choose. -
- Some tests are known to fail. Some are bugs that we have not fixed yet; - others are features that we haven't added yet (or may never add). In - QMTest, the result for such tests will be XFAIL (eXpected FAILure). In - this way, you can tell the difference between an expected and unexpected - failure. -
- The Programs tests have no such feature as of this time. If the test - passes, only warnings and other miscellaneous output will be generated. - If a test fails, a large <program> FAILED message will be - displayed. This will help you separate benign warnings from actual test - failures. -
-+The LLVM Nightly Testers +automatically check out an LLVM tree, build it, run the "nightly" +program test (described above), run all of the feature and regression tests, +and then delete the checked out tree. This tester is designed to ensure that +programs don't break as well as keep track of LLVM's progress over time.
+ ++If you'd like to set up an instance of the nightly tester to run on your +machine, take a look at the comments at the top of the utils/NightlyTester.pl +file. We usually run it from a crontab entry that looks ilke this: +
+ ++5 3 * * * $HOME/llvm/utils/NightlyTest.pl -parallel -enable-linscan ...CVSREPOSTORY... $HOME/buildtest-X86 $HOME/cvs/testresults-X86 ++ +
Or, you can create a shell script to encapsulate the running of the script. +The optimized x86 Linux nightly test is run from just such a script: +
+#!/bin/bash +BASE=/proj/work/llvm/nightlytest +export CVSROOT=:pserver:anon@llvm.cs.uiuc.edu:/var/cvs/llvm +export BUILDDIR=$BASE/build +export WEBDIR=$BASE/testresults +export LLVMGCCDIR=/proj/work/llvm/cfrontend/install +export PATH=/proj/install/bin:$LLVMGCCDIR/bin:$PATH +export LD_LIBRARY_PATH=/proj/install/lib +cd $BASE +cp /proj/work/llvm/llvm/utils/NightlyTest.pl . +nice ./NightlyTest.pl -nice -release -verbose -parallel -enable-linscan -noexternals ++ +
+Take a look at the NightlyTest.pl file to see what all of the flags and +strings do. If you start running the nightly tests, please let us know and +we'll link your page to the global tester page. Thanks! +
+ +
+