1 =================================
2 LLVM Testing Infrastructure Guide
3 =================================
5 Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya
14 This document is the reference manual for the LLVM testing
15 infrastructure. It documents the structure of the LLVM testing
16 infrastructure, the tools needed to use it, and how to add and run
22 In order to use the LLVM testing infrastructure, you will need all of
23 the software required to build LLVM, as well as
24 `Python <http://python.org>`_ 2.4 or later.
26 LLVM testing infrastructure organization
27 ========================================
29 The LLVM testing infrastructure contains two major categories of tests:
30 regression tests and whole programs. The regression tests are contained
31 inside the LLVM repository itself under ``llvm/test`` and are expected
32 to always pass -- they should be run before every commit.
34 The whole programs tests are referred to as the "LLVM test suite" (or
35 "test-suite") and are in the ``test-suite`` module in subversion. For
36 historical reasons, these tests are also referred to as the "nightly
37 tests" in places, which is less ambiguous than "test-suite" and remains
38 in use although we run them much more often than nightly.
43 The regression tests are small pieces of code that test a specific
44 feature of LLVM or trigger a specific bug in LLVM. They are usually
45 written in LLVM assembly language, but can be written in other languages
46 if the test targets a particular language front end (and the appropriate
47 ``--with-llvmgcc`` options were used at ``configure`` time of the
48 ``llvm`` module). These tests are driven by the 'lit' testing tool,
49 which is part of LLVM.
51 These code fragments are not complete programs. The code generated from
52 them is never executed to determine correct behavior.
54 These code fragment tests are located in the ``llvm/test`` directory.
56 Typically when a bug is found in LLVM, a regression test containing just
57 enough code to reproduce the problem should be written and placed
58 somewhere underneath this directory. In most cases, this will be a small
59 piece of LLVM assembly language code, often distilled from an actual
60 application or benchmark.
65 The test suite contains whole programs, which are pieces of code which
66 can be compiled and linked into a stand-alone program that can be
67 executed. These programs are generally written in high level languages
70 These programs are compiled using a user specified compiler and set of
71 flags, and then executed to capture the program output and timing
72 information. The output of these programs is compared to a reference
73 output to ensure that the program is being compiled correctly.
75 In addition to compiling and executing programs, whole program tests
76 serve as a way of benchmarking LLVM performance, both in terms of the
77 efficiency of the programs generated as well as the speed with which
78 LLVM compiles, optimizes, and generates code.
80 The test-suite is located in the ``test-suite`` Subversion module.
82 Debugging Information tests
83 ---------------------------
85 The test suite contains tests to check quality of debugging information.
86 The test are written in C based languages or in LLVM assembly language.
88 These tests are compiled and run under a debugger. The debugger output
89 is checked to validate of debugging information. See README.txt in the
90 test suite for more information . This test suite is located in the
91 ``debuginfo-tests`` Subversion module.
96 The tests are located in two separate Subversion modules. The
97 regressions tests are in the main "llvm" module under the directory
98 ``llvm/test`` (so you get these tests for free with the main llvm tree).
99 Use "make check-all" to run the regression tests after building LLVM.
101 The more comprehensive test suite that includes whole programs in C and
102 C++ is in the ``test-suite`` module. See ```test-suite``
103 Quickstart <#testsuitequickstart>`_ for more information on running
109 To run all of the LLVM regression tests, use master Makefile in the
110 ``llvm/test`` directory:
122 If you have `Clang <http://clang.llvm.org/>`_ checked out and built, you
123 can run the LLVM and Clang tests simultaneously using:
131 To run the tests with Valgrind (Memcheck by default), just append
132 ``VG=1`` to the commands above, e.g.:
138 To run individual tests or subsets of tests, you can use the 'llvm-lit'
139 script which is built as part of LLVM. For example, to run the
140 'Integer/BitPacked.ll' test by itself you can run:
144 % llvm-lit ~/llvm/test/Integer/BitPacked.ll
146 or to run all of the ARM CodeGen tests:
150 % llvm-lit ~/llvm/test/CodeGen/ARM
152 For more information on using the 'lit' tool, see 'llvm-lit --help' or
155 Debugging Information tests
156 ---------------------------
158 To run debugging information tests simply checkout the tests inside
159 clang/test directory.
164 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
166 These tests are already set up to run as part of clang regression tests.
168 Regression test structure
169 =========================
171 The LLVM regression tests are driven by 'lit' and are located in the
172 ``llvm/test`` directory.
174 This directory contains a large array of small tests that exercise
175 various features of LLVM and to ensure that regressions do not occur.
176 The directory is broken into several sub-directories, each focused on a
177 particular area of LLVM. A few of the important ones are:
179 - ``Analysis``: checks Analysis passes.
180 - ``Archive``: checks the Archive library.
181 - ``Assembler``: checks Assembly reader/writer functionality.
182 - ``Bitcode``: checks Bitcode reader/writer functionality.
183 - ``CodeGen``: checks code generation and each target.
184 - ``Features``: checks various features of the LLVM language.
185 - ``Linker``: tests bitcode linking.
186 - ``Transforms``: tests each of the scalar, IPO, and utility transforms
187 to ensure they make the right transformations.
188 - ``Verifier``: tests the IR verifier.
190 Writing new regression tests
191 ----------------------------
193 The regression test structure is very simple, but does require some
194 information to be set. This information is gathered via ``configure``
195 and is written to a file, ``lit.site.cfg`` in ``llvm/test``. The
196 ``llvm/test`` Makefile does this work for you.
198 In order for the regression tests to work, each directory of tests must
199 have a ``lit.local.cfg`` file. Lit looks for this file to determine how
200 to run the tests. This file is just Python code and thus is very
201 flexible, but we've standardized it for the LLVM regression tests. If
202 you're adding a directory of tests, just copy ``lit.local.cfg`` from
203 another directory to get running. The standard ``lit.local.cfg`` simply
204 specifies which files to look in for tests. Any directory that contains
205 only directories does not need the ``lit.local.cfg`` file. Read the `Lit
206 documentation <http://llvm.org/cmds/lit.html>`_ for more information.
208 The ``llvm-runtests`` function looks at each file that is passed to it
209 and gathers any lines together that match "RUN:". These are the "RUN"
210 lines that specify how the test is to be run. So, each test script must
211 contain RUN lines if it is to do anything. If there are no RUN lines,
212 the ``llvm-runtests`` function will issue an error and the test will
215 RUN lines are specified in the comments of the test program using the
216 keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
217 to execute. Together, these lines form the "script" that
218 ``llvm-runtests`` executes to run the test case. The syntax of the RUN
219 lines is similar to a shell's syntax for pipelines including I/O
220 redirection and variable substitution. However, even though these lines
221 may *look* like a shell script, they are not. RUN lines are interpreted
222 directly by the Tcl ``exec`` command. They are never executed by a
223 shell. Consequently the syntax differs from normal shell script syntax
224 in a few ways. You can specify as many RUN lines as needed.
226 lit performs substitution on each RUN line to replace LLVM tool names
227 with the full paths to the executable built for each tool (in
228 $(LLVM\_OBJ\_ROOT)/$(BuildMode)/bin). This ensures that lit does not
229 invoke any stray LLVM tools in the user's path during testing.
231 Each RUN line is executed on its own, distinct from other lines unless
232 its last character is ``\``. This continuation character causes the RUN
233 line to be concatenated with the next one. In this way you can build up
234 long pipelines of commands without making huge line lengths. The lines
235 ending in ``\`` are concatenated until a RUN line that doesn't end in
236 ``\`` is found. This concatenated set of RUN lines then constitutes one
237 execution. Tcl will substitute variables and arrange for the pipeline to
238 be executed. If any process in the pipeline fails, the entire line (and
239 test case) fails too.
241 Below is an example of legal RUN lines in a ``.ll`` file:
245 ; RUN: llvm-as < %s | llvm-dis > %t1
246 ; RUN: llvm-dis < %s.bc-13 > %t2
249 As with a Unix shell, the RUN: lines permit pipelines and I/O
250 redirection to be used. However, the usage is slightly different than
251 for Bash. To check what's legal, see the documentation for the `Tcl
252 exec <http://www.tcl.tk/man/tcl8.5/TclCmd/exec.htm#M2>`_ command and the
253 `tutorial <http://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html>`_. The
254 major differences are:
256 - You can't do ``2>&1``. That will cause Tcl to write to a file named
257 ``&1``. Usually this is done to get stderr to go through a pipe. You
258 can do that in tcl with ``|&`` so replace this idiom:
259 ``... 2>&1 | grep`` with ``... |& grep``
260 - You can only redirect to a file, not to another descriptor and not
261 from a here document.
262 - tcl supports redirecting to open files with the @ syntax but you
263 shouldn't use that here.
265 There are some quoting rules that you must pay attention to when writing
266 your RUN lines. In general nothing needs to be quoted. Tcl won't strip
267 off any quote characters so they will get passed to the invoked program.
272 ... | grep 'find this string'
274 This will fail because the ' characters are passed to grep. This would
275 instruction grep to look for ``'find`` in the files ``this`` and
276 ``string'``. To avoid this use curly braces to tell Tcl that it should
277 treat everything enclosed as one value. So our example would become:
281 ... | grep {find this string}
283 Additionally, the characters ``[`` and ``]`` are treated specially by
284 Tcl. They tell Tcl to interpret the content as a command to execute.
285 Since these characters are often used in regular expressions this can
286 have disastrous results and cause the entire test run in a directory to
287 fail. For example, a common idiom is to look for some basicblock number:
293 This, however, will cause Tcl to fail because its going to try to
294 execute a program named "2-8". Instead, what you want is this:
298 ... | grep {bb\[2-8\]}
300 Finally, if you need to pass the ``\`` character down to a program, then
301 it must be doubled. This is another Tcl special character. So, suppose
308 This will fail to match what you want (a pointer to i32). First, the
309 ``'`` do not get stripped off. Second, the ``\`` gets stripped off by
310 Tcl so what grep sees is: ``'i32*'``. That's not likely to match
311 anything. To resolve this you must use ``\\`` and the ``{}``, like this:
317 If your system includes GNU ``grep``, make sure that ``GREP_OPTIONS`` is
318 not set in your environment. Otherwise, you may get invalid results
319 (both false positives and false negatives).
321 The FileCheck utility
322 ---------------------
324 A powerful feature of the RUN: lines is that it allows any arbitrary
325 commands to be executed as part of the test harness. While standard
326 (portable) unix tools like 'grep' work fine on run lines, as you see
327 above, there are a lot of caveats due to interaction with Tcl syntax,
328 and we want to make sure the run lines are portable to a wide range of
329 systems. Another major problem is that grep is not very good at checking
330 to verify that the output of a tools contains a series of different
331 output in a specific order. The FileCheck tool was designed to help with
334 FileCheck (whose basic command line arguments are described in `the
335 FileCheck man page <http://llvm.org/cmds/FileCheck.html>`_ is designed
336 to read a file to check from standard input, and the set of things to
337 verify from a file specified as a command line argument. A simple
338 example of using FileCheck from a RUN line looks like this:
342 ; RUN: llvm-as < %s | llc -march=x86-64 | FileCheck %s
344 This syntax says to pipe the current file ("%s") into llvm-as, pipe that
345 into llc, then pipe the output of llc into FileCheck. This means that
346 FileCheck will be verifying its standard input (the llc output) against
347 the filename argument specified (the original .ll file specified by
348 "%s"). To see how this works, let's look at the rest of the .ll file
349 (after the RUN line):
353 define void @sub1(i32* %p, i32 %v) {
357 %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v)
361 define void @inc4(i64* %p) {
365 %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1)
369 Here you can see some "CHECK:" lines specified in comments. Now you can
370 see how the file is piped into llvm-as, then llc, and the machine code
371 output is what we are verifying. FileCheck checks the machine code
372 output to verify that it matches what the "CHECK:" lines specify.
374 The syntax of the CHECK: lines is very simple: they are fixed strings
375 that must occur in order. FileCheck defaults to ignoring horizontal
376 whitespace differences (e.g. a space is allowed to match a tab) but
377 otherwise, the contents of the CHECK: line is required to match some
378 thing in the test file exactly.
380 One nice thing about FileCheck (compared to grep) is that it allows
381 merging test cases together into logical groups. For example, because
382 the test above is checking for the "sub1:" and "inc4:" labels, it will
383 not match unless there is a "subl" in between those labels. If it
384 existed somewhere else in the file, that would not count: "grep subl"
385 matches if subl exists anywhere in the file.
387 The FileCheck -check-prefix option
388 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
390 The FileCheck -check-prefix option allows multiple test configurations
391 to be driven from one .ll file. This is useful in many circumstances,
392 for example, testing different architectural variants with llc. Here's a
397 ; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \
398 ; RUN: | FileCheck %s -check-prefix=X32
399 ; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \
400 ; RUN: | FileCheck %s -check-prefix=X64
402 define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind {
403 %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1
406 ; X32: pinsrd $1, 4(%esp), %xmm0
409 ; X64: pinsrd $1, %edi, %xmm0
412 In this case, we're testing that we get the expected code generation
413 with both 32-bit and 64-bit code generation.
415 The "CHECK-NEXT:" directive
416 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
418 Sometimes you want to match lines and would like to verify that matches
419 happen on exactly consecutive lines with no other lines in between them.
420 In this case, you can use CHECK: and CHECK-NEXT: directives to specify
421 this. If you specified a custom check prefix, just use "<PREFIX>-NEXT:".
422 For example, something like this works as you'd expect:
426 define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) {
427 %tmp3 = load <2 x double>* %A, align 16
428 %tmp7 = insertelement <2 x double> undef, double %B, i32 0
429 %tmp9 = shufflevector <2 x double> %tmp3,
431 <2 x i32> < i32 0, i32 2 >
432 store <2 x double> %tmp9, <2 x double>* %r, align 16
436 ; CHECK: movl 8(%esp), %eax
437 ; CHECK-NEXT: movapd (%eax), %xmm0
438 ; CHECK-NEXT: movhpd 12(%esp), %xmm0
439 ; CHECK-NEXT: movl 4(%esp), %eax
440 ; CHECK-NEXT: movapd %xmm0, (%eax)
444 CHECK-NEXT: directives reject the input unless there is exactly one
445 newline between it an the previous directive. A CHECK-NEXT cannot be the
446 first directive in a file.
448 The "CHECK-NOT:" directive
449 ^^^^^^^^^^^^^^^^^^^^^^^^^^
451 The CHECK-NOT: directive is used to verify that a string doesn't occur
452 between two matches (or the first match and the beginning of the file).
453 For example, to verify that a load is removed by a transformation, a
454 test like this can be used:
458 define i8 @coerce_offset0(i32 %V, i32* %P) {
459 store i32 %V, i32* %P
461 %P2 = bitcast i32* %P to i8*
462 %P3 = getelementptr i8* %P2, i32 2
466 ; CHECK: @coerce_offset0
471 FileCheck Pattern Matching Syntax
472 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
474 The CHECK: and CHECK-NOT: directives both take a pattern to match. For
475 most uses of FileCheck, fixed string matching is perfectly sufficient.
476 For some things, a more flexible form of matching is desired. To support
477 this, FileCheck allows you to specify regular expressions in matching
478 strings, surrounded by double braces: **{{yourregex}}**. Because we want
479 to use fixed string matching for a majority of what we do, FileCheck has
480 been designed to support mixing and matching fixed string matching with
481 regular expressions. This allows you to write things like this:
485 ; CHECK: movhpd {{[0-9]+}}(%esp), {{%xmm[0-7]}}
487 In this case, any offset from the ESP register will be allowed, and any
488 xmm register will be allowed.
490 Because regular expressions are enclosed with double braces, they are
491 visually distinct, and you don't need to use escape characters within
492 the double braces like you would in C. In the rare case that you want to
493 match double braces explicitly from the input, you can use something
494 ugly like **{{[{][{]}}** as your pattern.
499 It is often useful to match a pattern and then verify that it occurs
500 again later in the file. For codegen tests, this can be useful to allow
501 any register, but verify that that register is used consistently later.
502 To do this, FileCheck allows named variables to be defined and
503 substituted into patterns. Here is a simple example:
508 ; CHECK: notw [[REGISTER:%[a-z]+]]
509 ; CHECK: andw {{.*}}[[REGISTER]]
511 The first check line matches a regex (``%[a-z]+``) and captures it into
512 the variables "REGISTER". The second line verifies that whatever is in
513 REGISTER occurs later in the file after an "andw". FileCheck variable
514 references are always contained in ``[[ ]]`` pairs, are named, and their
515 names can be formed with the regex "``[a-zA-Z][a-zA-Z0-9]*``". If a
516 colon follows the name, then it is a definition of the variable, if not,
519 FileCheck variables can be defined multiple times, and uses always get
520 the latest value. Note that variables are all read at the start of a
521 "CHECK" line and are all defined at the end. This means that if you have
522 something like "``CHECK: [[XYZ:.*]]x[[XYZ]]``" that the check line will
523 read the previous value of the XYZ variable and define a new one after
524 the match is performed. If you need to do something like this you can
525 probably take advantage of the fact that FileCheck is not actually
526 line-oriented when it matches, this allows you to define two separate
527 CHECK lines that match on the same line.
529 Variables and substitutions
530 ---------------------------
532 With a RUN line there are a number of substitutions that are permitted.
533 In general, any Tcl variable that is available in the ``substitute``
534 function (in ``test/lib/llvm.exp``) can be substituted into a RUN line.
535 To make a substitution just write the variable's name preceded by a $.
536 Additionally, for compatibility reasons with previous versions of the
537 test library, certain names can be accessed with an alternate syntax: a
538 % prefix. These alternates are deprecated and may go away in a future
541 Here are the available variable names. The alternate syntax is listed in
545 The full path to the test case's source. This is suitable for passing on
546 the command line as the input to an llvm tool.
548 ``%(line)``, ``%(line+<number>)``, ``%(line-<number>)``
549 The number of the line where this variable is used, with an optional
550 integer offset. This can be used in tests with multiple RUN: lines,
551 which reference test file's line numbers.
554 The source directory from where the "``make check``" was run.
557 The object directory that corresponds to the ``$srcdir``.
560 A partial path from the ``test`` directory that contains the
561 sub-directory that contains the test source being executed.
564 The root directory of the LLVM src tree.
567 The root directory of the LLVM object tree. This could be the same as
571 The path to the directory that contains the test case source. This is
572 for locating any supporting files that are not generated by the test,
573 but used by the test.
576 The path to a temporary file name that could be used for this test case.
577 The file name won't conflict with other test cases. You can append to it
578 if you need multiple temporaries. This is useful as the destination of
579 some redirected output.
581 ``target_triplet`` (``%target_triplet``)
582 The target triplet that corresponds to the current host machine (the one
583 running the test cases). This should probably be called "host".
586 This full link command used to link LLVM executables. This has all the
587 configured -I, -L and -l options.
589 ``shlibext`` (``%shlibext``)
590 The suffix for the host platforms share library (dll) files. This
591 includes the period as the first character.
593 To add more variables, two things need to be changed. First, add a line
594 in the ``test/Makefile`` that creates the ``site.exp`` file. This will
595 "set" the variable as a global in the site.exp file. Second, in the
596 ``test/lib/llvm.exp`` file, in the substitute proc, add the variable
597 name to the list of "global" declarations at the beginning of the proc.
598 That's it, the variable can then be used in test scripts.
603 To make RUN line writing easier, there are several shell scripts located
604 in the ``llvm/test/Scripts`` directory. This directory is in the PATH
605 when running tests, so you can just call these scripts using their name.
609 This script runs its arguments and then always returns 0. This is useful
610 in cases where the test needs to cause a tool to generate an error (e.g.
611 to check the error output). However, any program in a pipeline that
612 returns a non-zero result will cause the test to fail. This script
613 overcomes that issue and nicely documents that the test case is
614 purposefully ignoring the result code of the tool
616 This script runs its arguments and then inverts the result code from it.
617 Zero result codes become 1. Non-zero result codes become 0. This is
618 useful to invert the result of a grep. For example "not grep X" means
619 succeed only if you don't find X in the input.
621 Sometimes it is necessary to mark a test case as "expected fail" or
622 XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
623 on a line near the top of the file. This signals that the test case
624 should succeed if the test fails. Such test cases are counted separately
625 by the testing tool. To specify an expected fail, use the XFAIL keyword
626 in the comments of the test program followed by a colon and one or more
627 failure patterns. Each failure pattern can be either ``*`` (to specify
628 fail everywhere), or a part of a target triple (indicating the test
629 should fail on that platform), or the name of a configurable feature
630 (for example, ``loadable_module``). If there is a match, the test is
631 expected to fail. If not, the test is expected to succeed. To XFAIL
632 everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
639 To make the output more useful, the ``llvm_runtest`` function wil scan
640 the lines of the test case for ones that contain a pattern that matches
641 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
642 that is related to the test case. The number after "PR" specifies the
643 LLVM bugzilla number. When a PR number is specified, it will be used in
644 the pass/fail reporting. This is useful to quickly get some context when
647 Finally, any line that contains "END." will cause the special
648 interpretation of lines to terminate. This is generally done right after
649 the last RUN: line. This has two side effects:
651 (a) it prevents special interpretation of lines that are part of the test
652 program, not the instructions to the test case, and
654 (b) it speeds things up for really big test cases by avoiding
655 interpretation of the remainder of the file.
657 ``test-suite`` Overview
658 =======================
660 The ``test-suite`` module contains a number of programs that can be
661 compiled and executed. The ``test-suite`` includes reference outputs for
662 all of the programs, so that the output of the executed program can be
663 checked for correctness.
665 ``test-suite`` tests are divided into three types of tests: MultiSource,
666 SingleSource, and External.
668 - ``test-suite/SingleSource``
670 The SingleSource directory contains test programs that are only a
671 single source file in size. These are usually small benchmark
672 programs or small programs that calculate a particular value. Several
673 such programs are grouped together in each directory.
675 - ``test-suite/MultiSource``
677 The MultiSource directory contains subdirectories which contain
678 entire programs with multiple source files. Large benchmarks and
679 whole applications go here.
681 - ``test-suite/External``
683 The External directory contains Makefiles for building code that is
684 external to (i.e., not distributed with) LLVM. The most prominent
685 members of this directory are the SPEC 95 and SPEC 2000 benchmark
686 suites. The ``External`` directory does not contain these actual
687 tests, but only the Makefiles that know how to properly compile these
688 programs from somewhere else. When using ``LNT``, use the
689 ``--test-externals`` option to include these tests in the results.
691 ``test-suite`` Quickstart
692 -------------------------
694 The modern way of running the ``test-suite`` is focused on testing and
695 benchmarking complete compilers using the
696 `LNT <http://llvm.org/docs/lnt>`_ testing infrastructure.
698 For more information on using LNT to execute the ``test-suite``, please
699 see the `LNT Quickstart <http://llvm.org/docs/lnt/quickstart.html>`_
702 ``test-suite`` Makefiles
703 ------------------------
705 Historically, the ``test-suite`` was executed using a complicated setup
706 of Makefiles. The LNT based approach above is recommended for most
707 users, but there are some testing scenarios which are not supported by
708 the LNT approach. In addition, LNT currently uses the Makefile setup
709 under the covers and so developers who are interested in how LNT works
710 under the hood may want to understand the Makefile based setup.
712 For more information on the ``test-suite`` Makefile setup, please see
713 the `Test Suite Makefile Guide. <TestSuiteMakefileGuide.html>`_