- <!--===============================================================-->
- <div class="doc_section"><a name="overview">Overview</a></div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>
- This document is the reference manual for the LLVM test suite. It
- documents the structure of the LLVM test suite, the tools needed to
- use it, and how to add and run tests.
- </p>
- </div>
-
- <!--===============================================================-->
- <div class="doc_section"><a name="Requirements">Requirements</a></div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>
- In order to use the LLVM test suite, you will need all of the software
- required to build LLVM, plus the following:
- </p>
- <dl compact>
- <dt><A HREF="http://www.qmtest.com">QMTest</A></dt>
- <dd>The LLVM test suite uses QMTest to organize and
- run tests.</dd>
-
- <dt><A HREF="http://www.python.org">Python</A></dt>
- <dd>You will need a Python interpreter that works with
- QMTest. Python will need zlib and SAX support
- enabled.</dd>
- </dl>
- </div>
-
- <!--===============================================================-->
- <div class="doc_section"><a name="quick">Quick Start</a></div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>
- The tests are located in the LLVM source tree under the directory
- <tt>llvm/test</tt>. To run all of the tests in LLVM, use the Master
- Makefile in that directory:
- </p>
- <pre>
- % make -C llvm/test
- </pre>
-
- <p>
- To run only the code fragment tests (i.e. those that do basic testing of
- LLVM), run the tests organized by QMTest:
- </p>
-
- <pre>
- % make -C llvm/test qmtest
- </pre>
-
- <p>
- To run only the tests that compile and execute whole programs, run the
- Programs tests:
- </p>
-
- <pre>
- % make -C llvm/test/Programs
- </pre>
- </div>
-
- <!--===============================================================-->
- <div class="doc_section"><h2><a name="org">LLVM Test Suite
- Organization </a></div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>The LLVM test suite contains two major categories of tests: code
- fragments and whole programs.</p>
- </div>
-
- <div class="doc_subsection"><a name="codefragments">Code Fragments</a>
- </div>
-
- <div class="doc_text">
- <p>
- Code fragments are small pieces of code that test a specific
- feature of LLVM or trigger a specific bug in LLVM. They are
- usually written in LLVM assembly language, but can be
- written in other languages if the test targets a
- particular language front end.
- </p><p>
- Code fragments are not complete programs, and they are
- never executed to determine correct behavior.
- </p><p>
- The tests in the Features and
- Regression directories contain code fragments.
- </p>
- </div>
-
- <div class="doc_subsection"><a name="wholeprograms">Whole Programs</a>
- </div>
-
- <div class="doc_text">
- <p>
- Whole Programs are pieces of code which can be compiled and
- linked into a stand-alone program that can be executed. These
- programs are generally written in high level languages such as C
- or C++, but sometimes they are written straight in LLVM
- assembly.
- </p><p>
- These programs are compiled and then executed using several
- different methods (native compiler, LLVM C backend, LLVM JIT,
- LLVM native code generation, etc). The output of these programs
- is compared to ensure that LLVM is compiling the program
- correctly.
- </p><p>
- In addition to compiling and executing programs, whole program
- tests serve as a way of benchmarking LLVM performance, both in
- terms of the efficiency of the programs generated as well as the
- speed with which LLVM compiles, optimizes, and generates code.
- </p><p>
- The Programs directory contains all tests which compile and
- benchmark whole programs.
- </p>
- </div>
-
- <!--===============================================================-->
- <div class="doc_section"><h2><a name="tree">LLVM Test Suite Tree</a>
- </div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>Each type of test in the LLVM test suite has its own directory. The
- major subtrees of the test suite directory tree are as follows:</p>
-
- <ul>
- <li>Features
- <p>
- This directory contains sample codes that test various features
- of the LLVM language. These pieces of sample code are run
- through various assembler, disassembler, and optimizer passes.
- </p>
-
- <li>Regression
- <p>
- This directory contains regression tests for LLVM. When a bug
- is found in LLVM, a regression test containing just enough
- code to reproduce the problem should be written and placed
- somewhere underneath this directory. In most cases, this
- will be a small piece of LLVM assembly language code, often
- distilled from an actual application or benchmark.
- </p>
-
- <li>Programs
- <p>
- The Programs directory contains programs that can be compiled
- with LLVM and executed. These programs are compiled using the
- native compiler and various LLVM backends. The output from the
- program compiled with the native compiler is assumed correct;
- the results from the other programs are compared to the native
- program output and pass if they match.
- </p><p>
- In addition for testing correctness, the Programs directory
- also performs timing tests of various LLVM optimizations.
- It also records compilation times for the compilers and the
- JIT. This information can be used to compare the
- effectiveness of LLVM's optimizations and code generation.
- </p><p>
- The Programs directory is subdivided into several smaller
- subdirectories:
- </p>
-
- <ul>
- <li>Programs/SingleSource
- <p>
- The SingleSource directory contains test programs that
- are only a single source file in size. These are
- usually small benchmark programs or small programs that
- calculate a particular value. Several such programs are
- grouped together in each directory.
- </p>
-
- <li>Programs/MultiSource
- <p>
- The MultiSource directory contains subdirectories which
- contain entire programs with multiple source files.
- Large benchmarks and whole applications go here.
- </p>
-
- <li>Programs/External
- <p>
- The External directory contains Makefiles for building
- code that is external to (i.e. not distributed with)
- LLVM. The most prominent member of this directory is
- the SPEC 2000 benchmark suite. The presence and
- location of these external programs is configured by the
- LLVM <tt>configure</tt> script.
- </p>
- </ul>
-
- <p>
-
- <li>QMTest
- <p>
- This directory contains the QMTest information files. Inside
- this directory are QMTest administration files and the Python
- code that implements the LLVM test and database classes.
- </p>
- </ul>
- </div>
-
- <!--===============================================================-->
- <div class="doc_section"><h2><a name="qmstructure">QMTest Structure</a>
- </div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>
- The LLVM test suite is partially driven by QMTest and partially
- driven by GNU Make. Specifically, the Features and Regression tests
- are all driven by QMTest. The Programs directory is currently
- driven by a set of Makefiles.
- </p><p>
- The QMTest system needs to have several pieces of information
- available; these pieces of configuration information are known
- collectively as the "context" in QMTest parlance. Since the context
- for LLVM is relatively large, the master Makefile in llvm/test
- sets it for you.
- </p><p>
- The LLVM database class makes the subdirectories of llvm/test a
- QMTest test database. For each directory that contains tests driven by
- QMTest, it knows what type of test the source file is and how to run it.
- </p><p>
- Hence, the QMTest namespace is essentially what you see in the
- Feature and Regression directories, but there is some magic that
- the database class performs (as described below).
- </p><p>
- The QMTest namespace is currently composed of the following tests and
- test suites:
- </p>
-
- <ul>
- <li>Feature
- <p>
- These are the feature tests found in the Feature directory.
- They are broken up into the following categories:
- </p>
- <ul>
- <li>ad
- <p>
- Assembler/Disassembler tests. These tests verify that a
- piece of LLVM assembly language can be assembled into
- bytecode and then disassembled into the original
- assembly language code. It does this several times to
- ensure that assembled output can be disassembled and
- disassembler output can be assembled. It also verifies
- that the give assembly language file can be assembled
- correctly.
- </p>
-
- <li>opt
- <p>
- Optimizer tests. These tests verify that two of the
- optimizer passes completely optimize a program (i.e.
- after a single pass, they cannot optimize a program
- any further).
- </p>
-
- <li>mc
- <p>
- Machine code tests. These tests verify that the LLVM
- assembly language file can be translated into native
- assembly code.
- </p>
-
- <li>cc
- <p>
- C code tests. These tests verify that the specified
- LLVM assembly code can be converted into C source code
- using the C backend.
- </p>
- </ul>
-
- <p>
- The LLVM database class looks at every file in the Feature
- directory and creates a fake test hierarchy containing
- <tt>Feature.<testtype>.<testname></tt>. So, if you
- add an LLVM assembly language file to the Feature directory, it
- actually creates 5 new tests: assembler/disassembler, assembler,
- optimizer, machine code, and C code.
- </p>
-
- <li>Regression
- <p>
- These are the regression tests. There is one suite for each
- subdirectory of the Regression directory. If you add a new
- subdirectory there, you will need to modify, at least, the
- <tt>RegressionMap</tt> variable in <tt>QMTest/llvmdb.py</tt> so
- that QMTest knows how to run the tests in the new subdirectory.
- </p>
- </ul>
- </div>
-
- <!--===============================================================-->
- <div class="doc_section"><h2><a name="progstructure">Programs
- Structure</a></div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>
- As mentioned previously, the Programs tree in llvm/test provides three
- types of tests: MultiSource, SingleSource, and External. Each tree is
- then subdivided into several categories, including applications,
- benchmarks, regression tests, code that is strange grammatically, etc.
- These organizations should be relatively self explanatory.
- </p><p>
- In addition to the regular Programs tests, the Programs tree also
- provides a mechanism for compiling the programs in different ways. If
- the variable TEST is defined on the gmake command line, the test system
- will include a Makefile named <tt>TEST.<value of TEST
- variable>.Makefile</tt>. This Makefile can modify build rules to
- yield different results.
- </p><p>
- For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt>
- to create the nightly test reports. To run the nightly tests, run
- <tt>gmake TEST=nightly</tt>.
- </p><p>
- There are several TEST Makefiles available in the tree. Some of them
- are designed for internal LLVM research and will not work outside of the
- LLVM research group. They may still be valuable, however, as a guide to
- writing your own TEST Makefile for any optimization or analysis passes
- that you develop with LLVM.
- </p>
- </div>
-
- <!--===============================================================-->
- <div class="doc_section"><h2><a name="run">Running the LLVM Tests</a>
- </div>
- <!--===============================================================-->
-
- <div class="doc_text">
- <p>
- First, all tests are executed within the LLVM object directory tree.
- They <i>are not</i> executed inside of the LLVM source tree. This is
- because the test suite creates temporary files during execution.
- </p><p>
- The master Makefile in llvm/test is capable of running both the
- QMTest driven tests and the Programs tests. By default, it will run
- all of the tests.
- </p><p>
- To run only the QMTest driven tests, run <tt>make qmtest</tt> at the
- command line in llvm/tests. To run a specific qmtest, suffix the test
- name with ".t" when running make.
- </p><p>
- For example, to run the Regression.LLC tests, type
- <tt>make Regression.LLC.t</tt> in llvm/tests.
- </p><p>
- Note that the Makefiles in llvm/test/Features and llvm/test/Regression
- are gone. You must now use QMTest from the llvm/test directory to run
- them.
- </p><p>
- To run the Programs test, cd into the llvm/test/Programs directory and
- type <tt>make</tt>. Alternatively, you can type <tt>make
- TEST=<type> test</tt> to run one of the specialized tests in
- llvm/test/Programs/TEST.<type>.Makefile. For example, you could
- run the nightly tester tests using the following commands:
- </p>
-
- <pre>
- % cd llvm/test/Programs
- % make TEST=nightly test
- </pre>
-
- <p>
- Regardless of which test you're running, the results are printed on
- standard output and standard error. You can redirect these results to a
- file if you choose.
- </p><p>
- Some tests are known to fail. Some are bugs that we have not fixed yet;
- others are features that we haven't added yet (or may never add). In
- QMTest, the result for such tests will be XFAIL (eXpected FAILure). In
- this way, you can tell the difference between an expected and unexpected
- failure.
- </p><p>
- The Programs tests have no such feature as of this time. If the test
- passes, only warnings and other miscellaneous output will be generated.
- If a test fails, a large <program> FAILED message will be
- displayed. This will help you separate benign warnings from actual test
- failures.
- </p>
- </div>