| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> | |
| <html xmlns="http://www.w3.org/1999/xhtml"> | |
| <head> | |
| <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" /> | |
| <link href="style.css" rel="stylesheet" type="text/css" /> | |
| <title>LLDB Projects</title> | |
| </head> | |
| <body> | |
| <div class="www_title"> | |
| Project ideas for the <strong>LLDB</strong> Debugger | |
| </div> | |
| <div id="container"> | |
| <div id="content"> | |
| <!--#include virtual="sidebar.incl"--> | |
| <div id="middle"> | |
| <div class="post"> | |
| <h1 class ="postheader">Projects</h1> | |
| <div class="postcontent"> | |
| <p>The following is a mostly unordered set of the ideas for improvements | |
| to the LLDB debugger. Some are fairly deep, some would require less | |
| effort. | |
| </p> | |
| <ol> | |
| <li> | |
| Speed up type realization in lldb. | |
| <p> | |
| The type of problem I'm addressing here is the situation where you are | |
| debugging a large program (lldb built with debug clang/swift will do) | |
| and you go to print a simple expression, and lldb goes away for 30 seconds. | |
| When you sample it, it is always busily churning through all the CU's in the | |
| world looking for something. The problem isn't that looking for something in | |
| particular is slow, but rather that we somehow turned an bounded search | |
| (maybe a subtype of "std::string" into an unbounded search (all things with the | |
| name of that subtype.) Or didn't stop when we got a reasonable answer | |
| proximate to the context of the search, but let the search leak out globally. | |
| And quite likely there are other issues that I haven't guessed yet. | |
| But if you end up churning though 3 or 4 Gig of debug info, that's going to be slow | |
| no matter how well written your debug reader is... | |
| </p> | |
| <p> | |
| My guess is the work will be more in the general symbol lookup than in the DWARF | |
| parser in particular, but it may be a combination of both. | |
| </p> | |
| <p> | |
| As a user debugging a largish program, this is the most obvious lameness of lldb. | |
| </p> | |
| </li> | |
| <li> | |
| Symbol name completion in the expression parser. | |
| <p> | |
| This is the other obvious lameness of lldb. You can do: | |
| </p> | |
| <code><pre><tt> | |
| (lldb) frame var foo.b<TAB> | |
| </tt></pre></code> | |
| <p> | |
| and we will tell you it is "foo.bar". But you can't do that in the expression parser. | |
| This will require collaboration with the clang/swift folks to get the right extension | |
| points in the compiler. And whatever they are, lldb will need use them to tell the | |
| compiler about what names are available. It will be important to avoid the pitfalls | |
| of #1 where we wander into the entire DWARF world. | |
| </p> | |
| </li> | |
| <li> | |
| Make a high speed asynchronous communication channel to replace the gdb-remote protocol. | |
| <p> | |
| All lldb debugging nowadays is done by talking to a debug agent. We used the | |
| gdb-remote protocol because that is universal, and good enough, and you have | |
| to support it anyway since so many little devices & JTAG's and VM's etc | |
| support it. But it is really old, not terribly high performance, and | |
| can't really handle sending or receiving messages while the process is | |
| supposedly running. It should have compression built in, remove the hand-built | |
| checksums and rely on the robust communication protocols we always have nowadays, | |
| allow for out-of-order requests/replies, allow for reconnecting to a temporarily | |
| disconnected debug session, regularize all of the packet formatting into JSON or | |
| BSON or whatever while including a way to do large binary transfers. It must be | |
| possible to come up with something faster, and better tunable for the many | |
| communications pathways we end up supporting. | |
| </p> | |
| </li> | |
| <li> | |
| Fix local variable lookup in the lldb expression parser. | |
| <p> | |
| The injection of local variables into the clang expression parser is | |
| currently done incorrectly - it happens too late in the lookup. This results | |
| in namespace variables & functions, same named types and ivars shadowing | |
| locals when it should be the other way around. An attempt was made to fix | |
| this by manually inserting all the visible local variables into wrapper function | |
| in the expression text. This mostly gets the job done but that method | |
| means you have to realize all the types | |
| and locations of all local variables for even the simplest of expressions, and | |
| when run on large programs (e.g. lldb) it would cause unacceptable delays. And | |
| it was very fragile since an error in realizing any of the locals would cause | |
| all expressions run in that context to fail. We need to fix this by adjusting | |
| the points where name lookup calls out to lldb in clang. | |
| </p> | |
| </li> | |
| <li> | |
| Fix the event handling/process control machinery to support calling SB & Commands | |
| everywhere, and to support non-stop debugging | |
| <p> | |
| There is a fairly ad-hoc system to handle when it is safe to run SB API's and | |
| command line commands. This is actually a bit of a tricky problem, since we | |
| allow access to the command line and SB API from some funky places in lldb. | |
| The Operating System plugins are the most obvious instance, since they get | |
| run right after lldb is told by debugserver that the process has stopped, but | |
| before it has finished collating the information from the stop for presentation | |
| to the higher levels. But breakpoint callbacks have some of the same problems, | |
| and other things like the scripted stepping operations and any fancier extension | |
| points we want to add to the debugger are going to be hard to implement robustly | |
| till we work on a finer-grained and more explicit control over who gets to control | |
| the process state. | |
| <p> | |
| We also won't have any chance of supporting non-stop debugging - which is a useful | |
| mode for programs that have a lot of high-priority or real-time worker threads - | |
| until we get this sorted out. | |
| </p> | |
| </li> | |
| <li> | |
| Finish the language abstraction and remove all the unnecessary C++/clang-based API's | |
| <p> | |
| An important part of making lldb a more useful "debugger toolkit" as opposed to a | |
| C/C++/ObjC/Swift debugger is to have a clean abstraction for language support. | |
| We did most, but not all, of the physical separation. We need to finish that. | |
| And then by force of necessity the API's really look like the interface to a C++ | |
| type system with a few swift bits added on. How you would go about adding a new | |
| language is unclear and much more trouble than it is worth at present. But if | |
| we made this nice, we could add a lot of value to other language projects. | |
| </p> | |
| </li> | |
| <li> | |
| Add some syntax to generate data formatters from type definitions | |
| <p> | |
| Uses of the data formatters fall into two types. There are data formatters for | |
| types where the structure elements pretty much tell you how to present the data, | |
| you just need a little expression language to express how to turn them into what | |
| the user expects to see. Then there are the ones (like pretty much all our | |
| Foundation/AppKit/UIKit formatters) that use deep magic to figure out how the type | |
| is actually laid out. The latter are pretty much always going to have to be done by hand. | |
| </p> | |
| <p> | |
| But for the ones where the information is expressed in the fields, it would be great | |
| to have a way to express the instructions to produce summaries and children in some | |
| form you could embed next to the types and have the compiler produce a byte code | |
| form of the instructions and then make that available to lldb along with the library. | |
| This isn't as simple as having clang run over the headers and produce something from the | |
| types directly. After all, clang has no way of knowing that the interesting thing about | |
| a std::vector is the elements that you get by calling size (for the summary) and [] for | |
| the elements. But it shouldn't be hard to come up with a generic markup to express this. | |
| </p> | |
| </li> | |
| <li> | |
| Allow the expression parser to access dynamic type/data formatter information | |
| <p> | |
| This seems like a smaller one. The symptom is your object is Foo child of Bar, and in | |
| the Locals view you see all the fields of Foo, but because the static type of the object | |
| is Bar, you can't see any of the fields of Foo. But if you could get this working, | |
| you could hijack the mechanism to make the results of the value object summaries/synthetic | |
| children available to expressions. And if you can do that, you could add other properties | |
| to an object externally (through Python or some other extension point) and then have | |
| these also available in the expression parser. You could use this to express invariants | |
| for data structures, or other more advanced uses of types in the debugger. | |
| </p> | |
| <p> | |
| Another version of this is to allow access to synthetic children in the expression | |
| parser. Otherwise you end up in situations like: | |
| </p> | |
| <code><pre><tt> | |
| (lldb) print return_a_foo() | |
| (SomeVectorLikeType) $1 = { | |
| [0] = 0 | |
| [1] = 1 | |
| [2] = 2 | |
| [3] = 3 | |
| [4] = 4 | |
| } | |
| </tt></pre></code> | |
| <p> | |
| That's good but: | |
| </p> | |
| <code><pre><tt> | |
| (lldb) print return_a_foo()[2] | |
| </tt></pre></code> | |
| <p> | |
| fails because the expression parser doesn't know anything about the array-like nature of | |
| SomeVectorLikeType that it gets from the synthetic children. | |
| </p> | |
| </li> | |
| <li> | |
| Recover thread information lazily | |
| <p> | |
| LLDB stores all the user intentions for a thread in the ThreadPlans stored in | |
| the Thread class. That allows us to reliably implement a very natural model for | |
| users moving through a debug session. For example, if step-over stops at a breakpoint | |
| in an function in a younger region of the stack, continue will complete the step-over | |
| rather than having to manually step out. But that means that it is important that | |
| the Thread objects live as long as the Threads they represent. For programs with many | |
| threads, but only one that you are debugging, that makes stepping less efficient, | |
| since now you have to fetch the thread list on every step or stepping doesn't work | |
| correctly. This is especially an issue when the threads are provided by an | |
| Operating System plugin, where it may take non-trivial work to reconstruct the | |
| thread list. It would be better to fetch threads lazily but keep "unseen" threads in | |
| a holding area, and only retire them when we know we've fetched the whole thread list | |
| and ensured they are no longer alive. | |
| </p> | |
| </li> | |
| <li> | |
| Add an extension point in the breakpoint search machinery. | |
| <p> | |
| This would allow highly customizable, algorithmic breakpoint types, like "break on every | |
| use of some particular instruction, or instruction pattern, etc." | |
| </p> | |
| </li> | |
| <li> | |
| Make Python-backed commands first class citizens | |
| <p> | |
| As it stands, Python commands have no way to advertise their options. They are | |
| required to parse their arguments by hand. That leads to inconsistency, and more | |
| importantly means they can't take advantage of auto-generated help and command | |
| completion. This leaves python-backed commands feeling worse than built-in ones. | |
| </p> | |
| <p> | |
| As part of this job, it would also be great to hook automatically hook the "type" of an option value | |
| or argument (e.g. eArgTypeShlibName) to sensible default completers. You need to be able to | |
| over-ride this in more complicated scenarios (like in "break set" where the presence of | |
| a "-s" option limits the search for completion of a "-n" option.) But in common cases it is | |
| unnecessary busy-work to have to supply the completer AND the type. If this worked, then | |
| it would be easier for Python commands to also get correct completers. | |
| </p> | |
| </li> | |
| <li> | |
| Reimplement the command interpreter commands using the SB API | |
| <p> | |
| Currently, all the CommandObject::DoExecute methods are implemented | |
| using the lldb_private API's. That generally means that there's code | |
| that gets duplicated between the CommandObject and the SB API that does | |
| roughly the same thing. We would reduce this code duplication, present a | |
| single coherent face to the users of lldb, and keep | |
| ourselves more honest about what we need in the SB API's if we implemented | |
| the CommandObjects::DoExecute methods using the SB API's. | |
| </p> | |
| <p> | |
| BTW, it is only the way it was much easier to develop lldb if it had a functioning | |
| command-line early on. So we did that first, and developed the SB API's when lldb | |
| was more mature. There's no good technical reason to have the commands use the | |
| lldb_private API's. | |
| </p> | |
| </li> | |
| <li> | |
| Documentation and better examples | |
| <p> | |
| We need to put the lldb syntax docs in the tutorial somewhere that is more easily accessible. | |
| On suggestion is to add non-command based help to the help system, and then have a "help lldb" | |
| or "help syntax" type command with this info. Be nice if the non-command based help could | |
| be hierarchical so you could make topics. | |
| </p> | |
| <p> | |
| There's a fair bit of docs about the SB API's, but it is spotty. Some classes are | |
| well documented in the Python "help (lldb.SBWhatever)" and some are not. | |
| </p> | |
| <p> | |
| We need more conceptual docs. And we need more examples. And we could provide a | |
| clean pluggable example for using LLDB standalone from Python. The process_events.py | |
| is a start of this, but it just handles process events, and it is really a quick sketch | |
| not a polished expandable proto-tool. | |
| </p> | |
| </li> | |
| <li> | |
| Make a more accessible plugin architecture for lldb. | |
| <p> | |
| Right now, you can only use the Python or SB API's to extend an extant lldb. You can't | |
| implement any of the actual lldb Plugins as plugins. That means anybody that wants to | |
| add new Object file/Process/Language etc support has to build and distribute their own | |
| lldb. This is tricky because the API's the plugins use are currently not stable (and | |
| recently have been changing quite a lot.) We would have to define a subset of lldb_private | |
| that you could use, and some way of telling whether the plugins were compatible with | |
| the lldb. But long-term, making this sort of extension possible will make lldb more | |
| appealing for research and 3rd party uses. | |
| </p> | |
| </li> | |
| <li> | |
| Use instruction emulation to avoid the overhead of swapping trap and instruction for breakpoints | |
| <p> | |
| At present, breakpoints are implemented by inserting a trap instruction, then when the trap | |
| is hit, replace the trap with the actual instruction and single step. Then swap back and | |
| continue. This causes problems for read only text, and also means that no-stop debugging | |
| ust either stop all threads briefly to handle this two-step or risk missing some | |
| breakpoint hits. If you emulated the instruction and wrote back the results, you wouldn't | |
| have these problems, and it would also save a stop per breakpoint hit. Since we use breakpoints | |
| to implement stepping, this savings could be significant on slow connections. | |
| </p> | |
| </li> | |
| <li> | |
| Use the JIT to speed up conditional breakpoint evaluation | |
| <p> | |
| We already JIT and cache the conditional expressions for breakpoints for the C family of | |
| languages, so we aren't re-compiling every time you hit the breakpoint. And if we couldn't | |
| IR interpret the expression, we leave the JIT'ed code in place for reuse. But it would | |
| be even better if we could also insert the "stop or not" decision into the code at the | |
| breakpoint, so you would only actually stop the process when the condition was true. | |
| Greg's idea was that if you had a conditional breakpoint set when you started the | |
| debug session, Xcode could rebuild and insert enough no-ops that we could instrument | |
| the breakpoint site and call the conditional expression, and only trap if the conditional was true. | |
| </p> | |
| </li> | |
| <li> | |
| Broaden the idea in "target stop-hook" to cover more events in the debugger | |
| <p> | |
| Shared library loads, command execution, User directed memory/register reads and writes | |
| are all places where you would reasonably want to hook into the debugger. | |
| </p> | |
| </li> | |
| <li> | |
| Mock classes for testing | |
| <p> | |
| We need "ProcessMock" and "ObjectFileMock" and the like. These would be real | |
| plugin implementations for their underlying lldb classes, with the addition | |
| that you can prime them from some sort of text based input files. For classes | |
| that manage changes over time (like process) you would need to program the state | |
| at StopPoint 0, StopPoint 1, etc. These could then be used for testing reactions | |
| to complex threading | |
| problems & the like, and also for simulating hard-to-test environments (like bare | |
| board debugging). | |
| </p> | |
| </li> | |
| <li> | |
| A Bug-Trapper infrastructure | |
| <p> | |
| We very often have bugs that can't be reproduced locally. So having a bug-report-trapper | |
| that can gather enough information from the | |
| surroundings of a bug so that we can replay the session locally would be a big help | |
| tracking down issues in this situation. This is tricky because you | |
| can't necessarily require folks to leak information about their code in order to | |
| file bug reports. So not only will you have to figure out what state to gather, | |
| you're also going to have to anonymize it somehow. But we very often have bugs | |
| from people that can't reduce the problem to a simple test case and can't give us | |
| our code, and we often just can't help them as things stand now. Note that adding | |
| the ProcessMock would be a good first stage towards this, since you could make a ProcessMock | |
| creator/serializer from the current lldb state. | |
| </p> | |
| </li> | |
| <li> | |
| Expression parser needs syntax for "{symbol,type} A in CU B.cpp" etc. | |
| <p> | |
| Sometimes you need to specify non-visible or ambiguous types to the expression parser. | |
| We were planning to do $b_dot_cpp$A or something like. You might want to specify a | |
| static in a function, in a source file, or in a shared library. So the syntax should | |
| support all these. | |
| </p> | |
| </li> | |
| <li> | |
| Add a "testButDontAbort" style test to the UnitTest framework. | |
| <p> | |
| The way we use unittest now (maybe this is the only way it can work, I don't know) | |
| you can't report a real failure and continue with the test. That is appropriate | |
| in some cases: if I'm supposed to hit breakpoint A before I evaluate an expression, | |
| and don't hit breakpoint A, the test should fail. But it means that if I want to | |
| test five different expressions, I can either do it in one test, which is good | |
| because it means I only have to fire up one process, attach to it, and get it to | |
| a certain point. But it also means if the first test fails, the other four don't | |
| even get run. So though at first we wrote a bunch of test like this, as time went | |
| on we switched more to writing "one at a time" tests because they were more robust | |
| against a single failure. That makes the test suite run much more slowly. It | |
| would be great to add a "test_but_dont_abort" variant of the tests, then we could | |
| gang tests that all drive to the same place and do similar things. As an added | |
| benefit, it would allow us to be more thorough in writing tests, since each | |
| test would have lower costs. | |
| </p> | |
| </li> | |
| <li> | |
| Convert the dotest style tests to use lldbutil.run_to_source_breakpoint. | |
| <p> | |
| run_to_source_breakpoint & run_to_name_breakpoint provide a compact API that | |
| does in one line what the first 10 or 20 lines of most of the old tests now do by | |
| hand. Using these functions makes tests much more readable, and by centralizing | |
| common functionality will make maintaining the testsuites easier in the future. | |
| This is more of a finger exercise, and perhaps best implemented by a rule like: | |
| "If you touch a test case, and it isn't using run_to_source_breakpoint, please | |
| make it do so". | |
| </p> | |
| </li> | |
| <li> | |
| Unify Watchpoint's & Breakpoints. | |
| <p> | |
| Option handling isn't shared, and more importantly the PerformAction's have a lot | |
| of duplicated common code, most of which works less well on the Watchpoint side. | |
| </p> | |
| </li> | |
| <li> | |
| Reverse debugging. | |
| <p> | |
| This is kind of a holy grail, it's hard to support for complex apps (many | |
| threads, shared memory, etc.) But it would be SO nice to have... | |
| </p> | |
| </li> | |
| <li> | |
| Non-stop debugging. | |
| <p> | |
| By this I mean allowing some threads in the target program to run while stopping | |
| other threads. This is supported in name in lldb at present, but lldb makes the | |
| assumption "If I get a stop, I won't get another stop unless I actually run the | |
| program." in a bunch of places so getting it to work reliably will be some a | |
| good bit of work. And figuring out how to present this in the UI will also be tricky. | |
| </p> | |
| </li> | |
| <li> | |
| Fix and continue. | |
| <p> | |
| We did this in gdb without a real JIT. The implementation shouldn't be that hard, | |
| especially if you can build the executable for fix and continue. The tricky part is | |
| how to verify that the user can only do the kinds of fixes that are safe to do. | |
| No changing object sizes is easy to detect, but there were many more subtle changes | |
| (function you are fixing is on the stack...) that take more work to prevent. | |
| And then you have to explain these conditions the user in some helpful way. | |
| </p> | |
| </li> | |
| <li> | |
| Unified IR interpreter. | |
| <p> | |
| Currently IRInterpreter implements a portion of the LLVM IR, but it doesn't handle | |
| vector data types and there are plenty of instructions it also doesn't support. | |
| Conversely, lli supports most of LLVM's IR but it doesn't handle remote memory and | |
| its function calling support is very rudimentary. It would be useful to unify these | |
| and make the IR interpreter -- both for LLVM and LLDB -- better. An alternate strategy | |
| would be simply to JIT into the current process but have callbacks for non-stack memory | |
| access. | |
| </p> | |
| </li> | |
| </ol> | |
| </div> | |
| <div class="postfooter"></div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </body> | |
| </html> |