X-Git-Url: http://demsky.eecs.uci.edu/git/?a=blobdiff_plain;f=docs%2FTestingGuide.rst;h=3cfbb219e6509c8d488b789fd0f2cbddc5827a1f;hb=fe532525cc4912ec0d1b4e91fa0396122dd087b3;hp=d542402608d05c5b621d069123d44c924c818b8c;hpb=56537a5886be63d0ed8b635f34c2473d6d1496dd;p=oota-llvm.git diff --git a/docs/TestingGuide.rst b/docs/TestingGuide.rst index d542402608d..3cfbb219e65 100644 --- a/docs/TestingGuide.rst +++ b/docs/TestingGuide.rst @@ -2,9 +2,6 @@ LLVM Testing Infrastructure Guide ================================= -Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya -Lattner - .. contents:: :local: @@ -169,30 +166,19 @@ The LLVM regression tests are driven by :program:`lit` and are located in the This directory contains a large array of small tests that exercise various features of LLVM and to ensure that regressions do not occur. The directory is broken into several sub-directories, each focused on a -particular area of LLVM. A few of the important ones are: - -- ``Analysis``: checks Analysis passes. -- ``Archive``: checks the Archive library. -- ``Assembler``: checks Assembly reader/writer functionality. -- ``Bitcode``: checks Bitcode reader/writer functionality. -- ``CodeGen``: checks code generation and each target. -- ``Features``: checks various features of the LLVM language. -- ``Linker``: tests bitcode linking. -- ``Transforms``: tests each of the scalar, IPO, and utility transforms - to ensure they make the right transformations. -- ``Verifier``: tests the IR verifier. +particular area of LLVM. Writing new regression tests ---------------------------- The regression test structure is very simple, but does require some information to be set. This information is gathered via ``configure`` -and is written to a file, ``lit.site.cfg`` in ``llvm/test``. The -``llvm/test`` Makefile does this work for you. +and is written to a file, ``test/lit.site.cfg`` in the build directory. +The ``llvm/test`` Makefile does this work for you. In order for the regression tests to work, each directory of tests must -have a ``lit.local.cfg`` file. Lit looks for this file to determine how -to run the tests. This file is just Python code and thus is very +have a ``lit.local.cfg`` file. :program:`lit` looks for this file to determine +how to run the tests. This file is just Python code and thus is very flexible, but we've standardized it for the LLVM regression tests. If you're adding a directory of tests, just copy ``lit.local.cfg`` from another directory to get running. The standard ``lit.local.cfg`` simply @@ -200,28 +186,24 @@ specifies which files to look in for tests. Any directory that contains only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit documentation ` for more information. -The ``llvm-runtests`` function looks at each file that is passed to it -and gathers any lines together that match "RUN:". These are the "RUN" -lines that specify how the test is to be run. So, each test script must -contain RUN lines if it is to do anything. If there are no RUN lines, -the ``llvm-runtests`` function will issue an error and the test will -fail. +Each test file must contain lines starting with "RUN:" that tell :program:`lit` +how to run it. If there are no RUN lines, :program:`lit` will issue an error +while running a test. RUN lines are specified in the comments of the test program using the keyword ``RUN`` followed by a colon, and lastly the command (pipeline) -to execute. Together, these lines form the "script" that -``llvm-runtests`` executes to run the test case. The syntax of the RUN -lines is similar to a shell's syntax for pipelines including I/O -redirection and variable substitution. However, even though these lines -may *look* like a shell script, they are not. RUN lines are interpreted -directly by the Tcl ``exec`` command. They are never executed by a -shell. Consequently the syntax differs from normal shell script syntax -in a few ways. You can specify as many RUN lines as needed. - -lit performs substitution on each RUN line to replace LLVM tool names +to execute. Together, these lines form the "script" that :program:`lit` +executes to run the test case. The syntax of the RUN lines is similar to a +shell's syntax for pipelines including I/O redirection and variable +substitution. However, even though these lines may *look* like a shell +script, they are not. RUN lines are interpreted by :program:`lit`. +Consequently, the syntax differs from shell in a few ways. You can specify +as many RUN lines as needed. + +:program:`lit` performs substitution on each RUN line to replace LLVM tool names with the full paths to the executable built for each tool (in -$(LLVM\_OBJ\_ROOT)/$(BuildMode)/bin). This ensures that lit does not -invoke any stray LLVM tools in the user's path during testing. +``$(LLVM_OBJ_ROOT)/$(BuildMode)/bin)``. This ensures that :program:`lit` does +not invoke any stray LLVM tools in the user's path during testing. Each RUN line is executed on its own, distinct from other lines unless its last character is ``\``. This continuation character causes the RUN @@ -229,8 +211,8 @@ line to be concatenated with the next one. In this way you can build up long pipelines of commands without making huge line lengths. The lines ending in ``\`` are concatenated until a RUN line that doesn't end in ``\`` is found. This concatenated set of RUN lines then constitutes one -execution. Tcl will substitute variables and arrange for the pipeline to -be executed. If any process in the pipeline fails, the entire line (and +execution. :program:`lit` will substitute variables and arrange for the pipeline +to be executed. If any process in the pipeline fails, the entire line (and test case) fails too. Below is an example of legal RUN lines in a ``.ll`` file: @@ -241,103 +223,123 @@ Below is an example of legal RUN lines in a ``.ll`` file: ; RUN: llvm-dis < %s.bc-13 > %t2 ; RUN: diff %t1 %t2 -As with a Unix shell, the RUN: lines permit pipelines and I/O -redirection to be used. However, the usage is slightly different than -for Bash. To check what's legal, see the documentation for the `Tcl -exec `_ command and the -`tutorial `_. The -major differences are: - -- You can't do ``2>&1``. That will cause Tcl to write to a file named - ``&1``. Usually this is done to get stderr to go through a pipe. You - can do that in tcl with ``|&`` so replace this idiom: - ``... 2>&1 | grep`` with ``... |& grep`` -- You can only redirect to a file, not to another descriptor and not - from a here document. -- tcl supports redirecting to open files with the @ syntax but you - shouldn't use that here. +As with a Unix shell, the RUN lines permit pipelines and I/O +redirection to be used. There are some quoting rules that you must pay attention to when writing -your RUN lines. In general nothing needs to be quoted. Tcl won't strip -off any quote characters so they will get passed to the invoked program. -For example: +your RUN lines. In general nothing needs to be quoted. :program:`lit` won't +strip off any quote characters so they will get passed to the invoked program. +To avoid this use curly braces to tell :program:`lit` that it should treat +everything enclosed as one value. -.. code-block:: bash +In general, you should strive to keep your RUN lines as simple as possible, +using them only to run tools that generate textual output you can then examine. +The recommended way to examine output to figure out if the test passes is using +the :doc:`FileCheck tool `. *[The usage of grep in RUN +lines is deprecated - please do not send or commit patches that use it.]* - ... | grep 'find this string' +Fragile tests +------------- -This will fail because the ' characters are passed to grep. This would -instruction grep to look for ``'find`` in the files ``this`` and -``string'``. To avoid this use curly braces to tell Tcl that it should -treat everything enclosed as one value. So our example would become: +It is easy to write a fragile test that would fail spuriously if the tool being +tested outputs a full path to the input file. For example, :program:`opt` by +default outputs a ``ModuleID``: -.. code-block:: bash +.. code-block:: console - ... | grep {find this string} + $ cat example.ll + define i32 @main() nounwind { + ret i32 0 + } -Additionally, the characters ``[`` and ``]`` are treated specially by -Tcl. They tell Tcl to interpret the content as a command to execute. -Since these characters are often used in regular expressions this can -have disastrous results and cause the entire test run in a directory to -fail. For example, a common idiom is to look for some basicblock number: + $ opt -S /path/to/example.ll + ; ModuleID = '/path/to/example.ll' -.. code-block:: bash + define i32 @main() nounwind { + ret i32 0 + } - ... | grep bb[2-8] +``ModuleID`` can unexpetedly match against ``CHECK`` lines. For example: -This, however, will cause Tcl to fail because its going to try to -execute a program named "2-8". Instead, what you want is this: +.. code-block:: llvm -.. code-block:: bash + ; RUN: opt -S %s | FileCheck - ... | grep {bb\[2-8\]} + define i32 @main() nounwind { + ; CHECK-NOT: load + ret i32 0 + } -Finally, if you need to pass the ``\`` character down to a program, then -it must be doubled. This is another Tcl special character. So, suppose -you had: +This test will fail if placed into a ``download`` directory. -.. code-block:: bash +To make your tests robust, always use ``opt ... < %s`` in the RUN line. +:program:`opt` does not output a ``ModuleID`` when input comes from stdin. - ... | grep 'i32\*' +Platform-Specific Tests +----------------------- -This will fail to match what you want (a pointer to i32). First, the -``'`` do not get stripped off. Second, the ``\`` gets stripped off by -Tcl so what grep sees is: ``'i32*'``. That's not likely to match -anything. To resolve this you must use ``\\`` and the ``{}``, like this: +Whenever adding tests that require the knowledge of a specific platform, +either related to code generated, specific output or back-end features, +you must make sure to isolate the features, so that buildbots that +run on different architectures (and don't even compile all back-ends), +don't fail. -.. code-block:: bash +The first problem is to check for target-specific output, for example sizes +of structures, paths and architecture names, for example: + +* Tests containing Windows paths will fail on Linux and vice-versa. +* Tests that check for ``x86_64`` somewhere in the text will fail anywhere else. +* Tests where the debug information calculates the size of types and structures. - ... | grep {i32\\*} +Also, if the test rely on any behaviour that is coded in any back-end, it must +go in its own directory. So, for instance, code generator tests for ARM go +into ``test/CodeGen/ARM`` and so on. Those directories contain a special +``lit`` configuration file that ensure all tests in that directory will +only run if a specific back-end is compiled and available. -If your system includes GNU ``grep``, make sure that ``GREP_OPTIONS`` is -not set in your environment. Otherwise, you may get invalid results -(both false positives and false negatives). +For instance, on ``test/CodeGen/ARM``, the ``lit.local.cfg`` is: + +.. code-block:: python + + config.suffixes = ['.ll', '.c', '.cpp', '.test'] + targets = set(config.root.targets_to_build.split()) + if not 'ARM' in targets: + config.unsupported = True + +Other platform-specific tests are those that depend on a specific feature +of a specific sub-architecture, for example only to Intel chips that support ``AVX2``. + +For instance, ``test/CodeGen/X86/psubus.ll`` tests three sub-architecture +variants: + +.. code-block:: llvm + + ; RUN: llc -mcpu=core2 < %s | FileCheck %s -check-prefix=SSE2 + ; RUN: llc -mcpu=corei7-avx < %s | FileCheck %s -check-prefix=AVX1 + ; RUN: llc -mcpu=core-avx2 < %s | FileCheck %s -check-prefix=AVX2 + +And the checks are different: + +.. code-block:: llvm -The FileCheck utility ---------------------- + ; SSE2: @test1 + ; SSE2: psubusw LCPI0_0(%rip), %xmm0 + ; AVX1: @test1 + ; AVX1: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 + ; AVX2: @test1 + ; AVX2: vpsubusw LCPI0_0(%rip), %xmm0, %xmm0 -A powerful feature of the RUN: lines is that it allows any arbitrary -commands to be executed as part of the test harness. While standard -(portable) unix tools like 'grep' work fine on run lines, as you see -above, there are a lot of caveats due to interaction with Tcl syntax, -and we want to make sure the run lines are portable to a wide range of -systems. Another major problem is that grep is not very good at checking -to verify that the output of a tools contains a series of different -output in a specific order. The FileCheck tool was designed to help with -these problems. +So, if you're testing for a behaviour that you know is platform-specific or +depends on special features of sub-architectures, you must add the specific +triple, test with the specific FileCheck and put it into the specific +directory that will filter out all other architectures. -FileCheck is designed to read a file to check from standard input, and the set -of things to verify from a file specified as a command line argument. -FileCheck is described in :doc:`the FileCheck man page -`. Variables and substitutions --------------------------- With a RUN line there are a number of substitutions that are permitted. -In general, any Tcl variable that is available in the ``substitute`` -function (in ``test/lib/llvm.exp``) can be substituted into a RUN line. -To make a substitution just write the variable's name preceded by a $. +To make a substitution just write the variable's name preceded by a ``$``. Additionally, for compatibility reasons with previous versions of the test library, certain names can be accessed with an alternate syntax: a % prefix. These alternates are deprecated and may go away in a future @@ -348,15 +350,15 @@ parentheses. ``$test`` (``%s``) The full path to the test case's source. This is suitable for passing on - the command line as the input to an llvm tool. + the command line as the input to an LLVM tool. ``%(line)``, ``%(line+)``, ``%(line-)`` The number of the line where this variable is used, with an optional - integer offset. This can be used in tests with multiple RUN: lines, + integer offset. This can be used in tests with multiple RUN lines, which reference test file's line numbers. ``$srcdir`` - The source directory from where the "``make check``" was run. + The source directory from where the ``make check`` was run. ``objdir`` The object directory that corresponds to the ``$srcdir``. @@ -389,23 +391,18 @@ parentheses. ``link`` (``%link``) This full link command used to link LLVM executables. This has all the - configured -I, -L and -l options. + configured ``-I``, ``-L`` and ``-l`` options. ``shlibext`` (``%shlibext``) - The suffix for the host platforms share library (dll) files. This + The suffix for the host platforms shared library (DLL) files. This includes the period as the first character. -To add more variables, two things need to be changed. First, add a line -in the ``test/Makefile`` that creates the ``site.exp`` file. This will -"set" the variable as a global in the site.exp file. Second, in the -``test/lib/llvm.exp`` file, in the substitute proc, add the variable -name to the list of "global" declarations at the beginning of the proc. -That's it, the variable can then be used in test scripts. +To add more variables, look at ``test/lit.cfg``. Other Features -------------- -To make RUN line writing easier, there are several shell scripts located +To make RUN line writing easier, there are several helper scripts and programs in the ``llvm/test/Scripts`` directory. This directory is in the PATH when running tests, so you can just call these scripts using their name. For example: @@ -419,9 +416,7 @@ For example: purposefully ignoring the result code of the tool ``not`` This script runs its arguments and then inverts the result code from it. - Zero result codes become 1. Non-zero result codes become 0. This is - useful to invert the result of a grep. For example "not grep X" means - succeed only if you don't find X in the input. + Zero result codes become 1. Non-zero result codes become 0. Sometimes it is necessary to mark a test case as "expected fail" or XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:`` @@ -441,7 +436,7 @@ line: ; XFAIL: darwin,sun -To make the output more useful, the ``llvm_runtest`` function wil scan +To make the output more useful, :program:`lit` will scan the lines of the test case for ones that contain a pattern that matches ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number that is related to the test case. The number after "PR" specifies the