1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
4 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
5 <link rel="stylesheet" href="llvm.css" type="text/css" media="screen" />
6 <title>LLVM Test Suite Guide</title>
11 <div class="doc_title">
16 <li><a href="#overview">Overview</a></li>
17 <li><a href="#Requirements">Requirements</a></li>
18 <li><a href="#quick">Quick Start</a></li>
19 <li><a href="#org">LLVM Test Suite Organization</a></li>
21 <li><a href="#codefragments">Code Fragments</a></li>
22 <li><a href="#wholeprograms">Whole Programs</a></li>
24 <li><a href="#tree">LLVM Test Suite Tree</a></li>
25 <li><a href="#qmstructure">QMTest Structure</a></li>
26 <li><a href="#progstructure">Programs Structure</a></li>
27 <li><a href="#run">Running the LLVM Tests</a></li>
28 <p><b>Written by John T. Criswell</b></p>
31 <!--===============================================================-->
32 <div class="doc_section"><a name="overview">Overview</a></div>
33 <!--===============================================================-->
35 <div class="doc_text">
37 This document is the reference manual for the LLVM test suite. It
38 documents the structure of the LLVM test suite, the tools needed to
39 use it, and how to add and run tests.
43 <!--===============================================================-->
44 <div class="doc_section"><a name="Requirements">Requirements</a></div>
45 <!--===============================================================-->
47 <div class="doc_text">
49 In order to use the LLVM test suite, you will need all of the software
50 required to build LLVM, plus the following:
53 <dt><A HREF="http://www.qmtest.com">QMTest</A></dt>
54 <dd>The LLVM test suite uses QMTest to organize and
57 <dt><A HREF="http://www.python.org">Python</A></dt>
58 <dd>You will need a Python interpreter that works with
59 QMTest. Python will need zlib and SAX support
64 <!--===============================================================-->
65 <div class="doc_section"><a name="quick">Quick Start</a></div>
66 <!--===============================================================-->
68 <div class="doc_text">
70 The tests are located in the LLVM source tree under the directory
71 <tt>llvm/test</tt>. To run all of the tests in LLVM, use the Master
72 Makefile in that directory:
79 To run only the code fragment tests (i.e. those that do basic testing of
80 LLVM), run the tests organized by QMTest:
84 % gmake -C llvm/test qmtest
88 To run only the tests that compile and execute whole programs, run the
93 % gmake -C llvm/test/Programs
97 <!--===============================================================-->
98 <div class="doc_section"><h2><a name="org">LLVM Test Suite
99 Organization </a></h2></div>
100 <!--===============================================================-->
102 <div class="doc_text">
103 <p>The LLVM test suite contains two major categories of tests: code
104 fragments and whole programs.</p>
107 <div class="doc_subsection"><a name="codefragments">Code Fragments</a>
110 <div class="doc_text">
112 Code fragments are small pieces of code that test a specific
113 feature of LLVM or trigger a specific bug in LLVM. They are
114 usually written in LLVM assembly language, but can be
115 written in other languages if the test targets a
116 particular language front end.
118 Code fragments are not complete programs, and they are
119 never executed to determine correct behavior.
121 The tests in the Features and
122 Regression directories contain code fragments.
126 <div class="doc_subsection"><a name="wholeprograms">Whole Programs</a>
129 <div class="doc_text">
131 Whole Programs are pieces of code which can be compiled and
132 linked into a stand-alone program that can be executed. These
133 programs are generally written in high level languages such as C
134 or C++, but sometimes they are written straight in LLVM
137 These programs are compiled and then executed using several
138 different methods (native compiler, LLVM C backend, LLVM JIT,
139 LLVM native code generation, etc). The output of these programs
140 is compared to ensure that LLVM is compiling the program
143 In addition to compiling and executing programs, whole program
144 tests serve as a way of benchmarking LLVM performance, both in
145 terms of the efficiency of the programs generated as well as the
146 speed with which LLVM compiles, optimizes, and generates code.
148 The Programs directory contains all tests which compile and
149 benchmark whole programs.
153 <!--===============================================================-->
154 <div class="doc_section"><h2><a name="tree">LLVM Test Suite Tree</a>
156 <!--===============================================================-->
158 <div class="doc_text">
159 <p>Each type of test in the LLVM test suite has its own directory. The
160 major subtrees of the test suite directory tree are as follows:</p>
165 This directory contains sample codes that test various features
166 of the LLVM language. These pieces of sample code are run
167 through various assembler, disassembler, and optimizer passes.
172 This directory contains regression tests for LLVM. When a bug
173 is found in LLVM, a regression test containing just enough
174 code to reproduce the problem should be written and placed
175 somewhere underneath this directory. In most cases, this
176 will be a small piece of LLVM assembly language code, often
177 distilled from an actual application or benchmark.
182 The Programs directory contains programs that can be compiled
183 with LLVM and executed. These programs are compiled using the
184 native compiler and various LLVM backends. The output from the
185 program compiled with the native compiler is assumed correct;
186 the results from the other programs are compared to the native
187 program output and pass if they match.
189 In addition for testing correctness, the Programs directory
190 also performs timing tests of various LLVM optimizations.
191 It also records compilation times for the compilers and the
192 JIT. This information can be used to compare the
193 effectiveness of LLVM's optimizations and code generation.
195 The Programs directory is subdivided into several smaller
200 <li>Programs/SingleSource
202 The SingleSource directory contains test programs that
203 are only a single source file in size. These are
204 usually small benchmark programs or small programs that
205 calculate a particular value. Several such programs are
206 grouped together in each directory.
209 <li>Programs/MultiSource
211 The MultiSource directory contains subdirectories which
212 contain entire programs with multiple source files.
213 Large benchmarks and whole applications go here.
216 <li>Programs/External
218 The External directory contains Makefiles for building
219 code that is external to (i.e. not distributed with)
220 LLVM. The most prominent member of this directory is
221 the SPEC 2000 benchmark suite. The presence and
222 location of these external programs is configured by the
223 LLVM <tt>configure</tt> script.
231 This directory contains the QMTest information files. Inside
232 this directory are QMTest administration files and the Python
233 code that implements the LLVM test and database classes.
238 <!--===============================================================-->
239 <div class="doc_section"><h2><a name="qmstructure">QMTest Structure</a>
241 <!--===============================================================-->
243 <div class="doc_text">
245 The LLVM test suite is partially driven by QMTest and partially
246 driven by GNU Make. Specifically, the Features and Regression tests
247 are all driven by QMTest. The Programs directory is currently
248 driven by a set of Makefiles.
250 The QMTest system needs to have several pieces of information
251 available; these pieces of configuration information are known
252 collectively as the "context" in QMTest parlance. Since the context
253 for LLVM is relatively large, the master Makefile in llvm/test
256 The LLVM database class makes the subdirectories of llvm/test a
257 QMTest test database. For each directory that contains tests driven by
258 QMTest, it knows what type of test the source file is and how to run it.
260 Hence, the QMTest namespace is essentially what you see in the
261 Feature and Regression directories, but there is some magic that
262 the database class performs (as described below).
264 The QMTest namespace is currently composed of the following tests and
271 These are the feature tests found in the Feature directory.
272 They are broken up into the following categories:
277 Assembler/Disassembler tests. These tests verify that a
278 piece of LLVM assembly language can be assembled into
279 bytecode and then disassembled into the original
280 assembly language code. It does this several times to
281 ensure that assembled output can be disassembled and
282 disassembler output can be assembled. It also verifies
283 that the give assembly language file can be assembled
289 Optimizer tests. These tests verify that two of the
290 optimizer passes completely optimize a program (i.e.
291 after a single pass, they cannot optimize a program
297 Machine code tests. These tests verify that the LLVM
298 assembly language file can be translated into native
304 C code tests. These tests verify that the specified
305 LLVM assembly code can be converted into C source code
311 The LLVM database class looks at every file in the Feature
312 directory and creates a fake test hierarchy containing
313 <tt>Feature.<testtype>.<testname></tt>. So, if you
314 add an LLVM assembly language file to the Feature directory, it
315 actually creates 5 new tests: assembler/disassembler, assembler,
316 optimizer, machine code, and C code.
321 These are the regression tests. There is one suite for each
322 subdirectory of the Regression directory. If you add a new
323 subdirectory there, you will need to modify, at least, the
324 <tt>RegressionMap</tt> variable in <tt>QMTest/llvmdb.py</tt> so
325 that QMTest knows how to run the tests in the new subdirectory.
330 <!--===============================================================-->
331 <div class="doc_section"><h2><a name="progstructure">Programs
333 <!--===============================================================-->
335 <div class="doc_text">
337 As mentioned previously, the Programs tree in llvm/test provides three
338 types of tests: MultiSource, SingleSource, and External. Each tree is
339 then subdivided into several categories, including applications,
340 benchmarks, regression tests, code that is strange grammatically, etc.
341 These organizations should be relatively self explanatory.
343 In addition to the regular Programs tests, the Programs tree also
344 provides a mechanism for compiling the programs in different ways. If
345 the variable TEST is defined on the gmake command line, the test system
346 will include a Makefile named <tt>TEST.<value of TEST
347 variable>.Makefile</tt>. This Makefile can modify build rules to
348 yield different results.
350 For example, the LLVM nightly tester uses <tt>TEST.nightly.Makefile</tt>
351 to create the nightly test reports. To run the nightly tests, run
352 <tt>gmake TEST=nightly</tt>.
354 There are several TEST Makefiles available in the tree. Some of them
355 are designed for internal LLVM research and will not work outside of the
356 LLVM research group. They may still be valuable, however, as a guide to
357 writing your own TEST Makefile for any optimization or analysis passes
358 that you develop with LLVM.
362 <!--===============================================================-->
363 <div class="doc_section"><h2><a name="run">Running the LLVM Tests</a>
365 <!--===============================================================-->
367 <div class="doc_text">
369 First, all tests are executed within the LLVM object directory tree.
370 They <i>are not</i> executed inside of the LLVM source tree. This is
371 because the test suite creates temporary files during execution.
373 The master Makefile in llvm/test is capable of running both the
374 QMTest driven tests and the Programs tests. By default, it will run
377 To run only the QMTest driven tests, run <tt>gmake qmtest</tt> at the
378 command line in llvm/tests. To run a specific qmtest, suffix the test
379 name with ".t" when running gmake.
381 For example, to run the Regression.LLC tests, type
382 <tt>gmake Regression.LLC.t</tt> in llvm/tests.
384 Note that the Makefiles in llvm/test/Features and llvm/test/Regression
385 are gone. You must now use QMTest from the llvm/test directory to run
388 To run the Programs test, cd into the llvm/test/Programs directory and
389 type <tt>gmake</tt>. Alternatively, you can type <tt>gmake
390 TEST=<type> test</tt> to run one of the specialized tests in
391 llvm/test/Programs/TEST.<type>.Makefile. For example, you could
392 run the nightly tester tests using the following commands:
396 % cd llvm/test/Programs
397 % gmake TEST=nightly test
401 Regardless of which test you're running, the results are printed on
402 standard output and standard error. You can redirect these results to a
405 Some tests are known to fail. Some are bugs that we have not fixed yet;
406 others are features that we haven't added yet (or may never add). In
407 QMTest, the result for such tests will be XFAIL (eXpected FAILure). In
408 this way, you can tell the difference between an expected and unexpected
411 The Programs tests have no such feature as of this time. If the test
412 passes, only warnings and other miscellaneous output will be generated.
413 If a test fails, a large <program> FAILED message will be
414 displayed. This will help you separate benign warnings from actual test
419 <!-- *********************************************************************** -->
422 <address>John T. Criswell</address>
423 <a href="http://llvm.cs.uiuc.edu">The LLVM Compiler Infrastructure</a>
425 Last modified: $Date$