| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> | |
| <html xmlns="http://www.w3.org/1999/xhtml"> | |
| <head> | |
| <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> | |
| <link href="style.css" rel="stylesheet" type="text/css" /> | |
| <title>Testing LLDB</title> | |
| </head> | |
| <body> | |
| <div class="www_title"> | |
| The <strong>LLDB</strong> Debugger | |
| </div> | |
| <div id="container"> | |
| <div id="content"> | |
| <!--#include virtual="sidebar.incl"--> | |
| <div id="middle"> | |
| <div class="post"> | |
| <h1 class="postheader">Testing LLDB</h1> | |
| <div class="postcontent"> | |
| <p> | |
| The LLDB test suite consists of Python scripts located under the | |
| <tt>test</tt> directory. Each script contains a number of test cases and is usually | |
| accompanied by a C (C++, ObjC, etc.) source file. Each test first compiles the | |
| source file and then uses LLDB to debug the resulting executable. The tests verify | |
| both the LLDB command line interface and the scripting API. | |
| </p> | |
| </div> | |
| <h1 class="postheader">Running tests</h1> | |
| <div class="postcontent"> | |
| <h2>Running the full test suite</h2> | |
| <p> | |
| <strong>Windows Note</strong>: In the examples that follow, any invocations of <code>python</code> | |
| should be replaced with <code>python_d</code>, the debug interpreter, when running the test | |
| suite against a debug version of LLDB. | |
| </p> | |
| <p> | |
| The easiest way to run the LLDB test suite is to use the <tt>check-lldb</tt> build | |
| target. By default, the <tt>check-lldb</tt> target builds the test programs with | |
| the same compiler that was used to build LLDB. To build the tests with a different | |
| compiler, you can set the <strong>LLDB_TEST_C_COMPILER</strong> or the <strong>LLDB_TEST_CXX_COMPILER</strong> CMake variables. | |
| These variables are ignored unless the respective <strong>LLDB_TEST_USE_CUSTOM_C_COMPILER</strong> and | |
| <strong>LLDB_TEST_USE_CUSTOM_CXX_COMPILER</strong> are set to ON. | |
| It is possible to customize the architecture of the test binaries and compiler used by appending -A | |
| and -C options respectively to the CMake variable <strong>LLDB_TEST_USER_ARGS</strong>. For | |
| example, to test LLDB against 32-bit binaries | |
| built with a custom version of clang, do: | |
| </p> | |
| <code> | |
| <br />> cmake -DLLDB_TEST_USER_ARGS="-A i386 -C /path/to/custom/clang" -G Ninja | |
| <br />> ninja check-lldb | |
| </code> | |
| <p>Note that multiple -A and -C flags can be specified to <tt>LLDB_TEST_USER_ARGS</tt>.</p> | |
| <p>Note that on NetBSD you must export <tt>LD_LIBRARY_PATH=$PWD/lib</tt> in your environment. This is due to lack of | |
| the <tt>$ORIGIN</tt> linker feature.</p> | |
| <h2>Running a specific test or set of tests</h2> | |
| <p> | |
| In addition to running all the LLDB test suites with the "check-lldb" CMake target above, it is possible to | |
| run individual LLDB tests. For example, to run the test cases defined in TestInferiorCrashing.py, run: | |
| </p> | |
| <code> | |
| <br />> cd $lldb/test | |
| <br />> python dotest.py --executable <path-to-lldb> -p TestInferiorCrashing.py ../packages/Python/lldbsuite/test | |
| </code> | |
| <p> | |
| If the test is not specified by name (e.g. if you leave the <code>-p</code> argument off), LLDB will run all tests in | |
| that directory: | |
| </p> | |
| <code> | |
| <br />> python dotest.py --executable <path-to-lldb> functionalities/data-formatter | |
| </code> | |
| <p> | |
| Many more options that are available. To see a list of all of them, run: | |
| </p> | |
| <code> | |
| > python dotest.py -h | |
| </code> | |
| <p> | |
| The dotest.py script runs tests in parallel by default. | |
| To disable the parallel test running feature, use the | |
| <code>--no-multiprocess</code> flag. The number of | |
| concurrent tests is controlled by | |
| the <code>LLDB_TEST_THREADS</code> environment variable | |
| or the <code>--threads</code> command line parameter. | |
| The default value is the number of CPU cores on your | |
| system. | |
| </p> | |
| <p> | |
| The parallel test running feature will handle an | |
| additional <code>--test-subdir SUBDIR</code> arg. When | |
| specified, SUBDIR is relative to the root test directory | |
| and will limit all parallel test running to that | |
| subdirectory's tree of tests. | |
| </p> | |
| <p> | |
| The parallel test runner will run all tests within a | |
| given directory serially, but will run multiple | |
| directories concurrently. Thus, as a test writer, we | |
| provide serialized test run semantics within a | |
| directory. Note child directories are considered | |
| entirely separate, so two child directories could be | |
| running in parallel with a parent directory. | |
| </p> | |
| <h2>Running the test-suite remotely</h2> | |
| <p> | |
| Running the test-suite remotely is similar to the process of running a local test | |
| suite, but there are two things to have in mind: | |
| </p> | |
| <ul> | |
| <li> | |
| You must have the <code>lldb-server</code> running on the remote system, ready to | |
| accept multiple connections. For more information on how to setup remote | |
| debugging see the <a href="remote.html">Remote debugging</a> page. | |
| </li> | |
| <li> | |
| You must tell the test-suite how to connect to the remote system. This is | |
| achieved using the <code>--platform-name</code>, <code>--platform-url</code> and | |
| <code>--platform-working-dir</code> parameters to <code>dotest.py</code>. These | |
| parameters correspond to the <code>platform select</code> and <code>platform | |
| connect</code> LLDB commands. You will usually also need to specify the compiler and | |
| architecture for the remote system. | |
| </li> | |
| </ul> | |
| <p> | |
| Currently, running the remote test suite is supported only with | |
| <code>dotest.py</code> (or <code>dosep.py</code> with a single thread), but we | |
| expect this issue to be addressed in the near future. | |
| </p> | |
| </div> | |
| <div class="postfooter"></div> | |
| <h1 class="postheader">Debugging test failures</h1> | |
| <div class="postcontent"> | |
| <h2>Non-Windows platforms</h2> | |
| <p> | |
| On non-Windows platforms, you can use the <code>-d</code> option to <code>dotest.py</code> which will cause the script to wait | |
| for a while until a debugger is attached. | |
| </p> | |
| <h2>Windows</h2> | |
| <p> | |
| On Windows, it is strongly recommended to use <a href="https://github.com/Microsoft/PTVS/releases">Python Tools for Visual Studio</a> | |
| for debugging test failures. It can seamlessly step between native and managed code, which is very helpful when you need to step | |
| through the test itself, and then into the LLDB code that backs the operations the test is performing. A quick guide to getting | |
| started with PTVS is as follows: | |
| <ul> | |
| <li>Install PTVS</li> | |
| <li> | |
| Create a Visual Studio Project for the Python code. | |
| <ul> | |
| <li>Go to File -> New -> Project -> Python -> From Existing Python Code.</li> | |
| <li>Choose <code>llvm/tools/lldb</code> as the directory containing the Python code.</li> | |
| <li> | |
| When asked where to save the <code>.pyproj</code> file, choose the folder <code>llvm/tools/lldb/pyproj</code>. | |
| This is a special folder that is ignored by the <code>.gitignore</code> file, since it is not checked in. | |
| </li> | |
| </ul> | |
| </li> | |
| <li>Set <code>test/dotest.py</code> as the startup file</li> | |
| <li> | |
| Make sure there is a Python Environment installed for your distribution. For example, if you installed Python to | |
| <code>C:\Python35</code>, PTVS needs to know that this is the interpreter you want to use for running the test suite. | |
| <ul> | |
| <li>Go to Tools -> Options -> Python Tools -> Environment Options</li> | |
| <li>Click Add Environment, and enter <code>Python 3.5 Debug</code> for the name. Fill out the values correctly.</li> | |
| </ul> | |
| </li> | |
| <li> | |
| Configure the project to use this debug interpreter. | |
| <ul> | |
| <li>Right click the Project node in Solution Explorer</li> | |
| <li>In the <code>General</code> tab, Make sure <code>Python 3.5 Debug</code> is the selected Interpreter.</li> | |
| <li>In <code>Debug/Search Paths</code>, enter the path to your <code>ninja/lib/site-packages</code> directory.</li> | |
| <li> | |
| In <code>Debug/Environment Variables</code>, enter<br/> | |
| <code>VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\</code> | |
| </li> | |
| <li> | |
| If you want to enabled mixed mode debugging, check <code>Enable native code debugging</code> (this slows down debugging, | |
| so enable it only on an as-needed basis.) | |
| </li> | |
| </ul> | |
| </li> | |
| <li> | |
| Set the command line for the test suite to run. | |
| <ul> | |
| <li>Right click the project in solution explorer and choose the <code>Debug</code> tab.</li> | |
| <li>Enter the arguments to <code>dotest.py</code>. Note you must add <code>--no-multiprocess</code></li> | |
| <li> | |
| Example command options: | |
| <code> | |
| <br/># quiet mode | |
| <br/>-q | |
| <br />--arch=i686 | |
| <br /># Path to debug lldb.exe | |
| <br />--executable D:/src/llvmbuild/ninja/bin/lldb.exe | |
| <br /># Directory to store log files | |
| <br />-s D:/src/llvmbuild/ninja/lldb-test-traces | |
| <br />-u CXXFLAGS -u CFLAGS | |
| <br /># If a test crashes, show JIT debugging dialog. | |
| <br />--enable-crash-dialog | |
| <br /># Path to release clang.exe | |
| <br />-C d:\src\llvmbuild\ninja_release\bin\clang.exe | |
| <br /># Path to the particular test you want to debug. | |
| <br />-p TestPaths.py | |
| <br /># Root of test tree | |
| <br />D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test | |
| <br /># Required in order to be able to debug the test. | |
| <br />--no-multiprocess | |
| </code> | |
| </li> | |
| <li> | |
| As copy-pastable command line:<br/> | |
| <code> | |
| -q --arch=i686 --executable D:/src/llvmbuild/ninja/bin/lldb.exe -s D:/src/llvmbuild/ninja/lldb-test-traces | |
| -u CXXFLAGS -u CFLAGS --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe | |
| -p TestPaths.py D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --no-multiprocess | |
| </code> | |
| </li> | |
| </ul> | |
| </li> | |
| </ul> | |
| </p> | |
| </div> | |
| <div class="postfooter"></div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </body> | |
| </html> |