Carbide, CDT, EDC Update

Time for an update from our team here at Nokia working on Carbide 3.0, Eclipse CDT 7.0, and the new Eclipse Debugger for C/C++ (EDC):

EDC is now part of the regular CDT build. Many thanks to Doug Schaefer for getting this done. I hadn't done much with the CDT build before and wasn't making much progress, but Doug resolved a couple tricky issues, like pulling the TCF source in from Subversion. We still have a version of EDC in our CVS so the Carbide team can easily commit stuff, but I'm syncing it every few days.

Custom Variable Formatting in EDC: David committed the first phase of support for custom variable formatting. Using a new extension point a plug-in can alter the way data is shown in the Variables view during an EDC debug session. An extension can provide an alternative to the standard structure and content of any variable type.

We intend to provide formatters for some STL, Qt & Symbian data types. Also there are general purpose formatters for arrays and structs:

 
Unformatted C/C++ struct

Formatted C/C++ struct





Debug Snapshot Albums: A new feature in EDC is the ability to capture a snapshot of the state of your debug session for later review. A new Snapshot Album view shows the available albums of snapshots from prior debug sessions and lets you annotate their contents. Snapshot Albums can be launched to show a re-creation of the original debug session. Tim and I have been overhauling some of the initial work and getting the UI far enough along so that we can start usability testing soon. I'll write more about this when the work settles down a bit.

Plug & Play Debug Connectivity: We've made a lot of improvements in usability when starting a debug session on a phone. We started with a detailed analysis of all the options available when connecting to a phone, then narrowed them down into a new launch wizard that can start a debug session in just a few steps. This will make on-device debugging in Carbide a lot easier and I hope to apply some of the lessons learned when we overhaul the launch experience in CDT.

EDC Symbol Reader Overhaul: Ed & Steve are in the midst of an overhaul of the Dwarf reader in EDC. We originally wrote it for completeness and to have something to build the rest of the debugger around, but knew more work was required to get better performance and memory usage. This effort will likely be continuing for the next several weeks.

Community Contributions to EDC: People are starting to review the code, report bugs, and contribute patches. In particular John Cortell at Freescale has been improving the way EDC connects to TCF agents and providing a fresh perspective on everything from naming to API visibility.

That's it for now, I'm diving back into the code...

Eclipse Debugger for C/C++ (EDC)

I'm very happy to announce that the sources for EDC, the Eclipse Debugger for C/C++, are now in the Eclipse CVS repository. EDC is a new component of the CDT project that provides a complete architecture for C/C++ debugging within Eclipse/CDT.

The Carbide team started developing EDC after setting on a new course that would open up nearly all of our development tools to either the Eclipse or Symbian communities. We couldn't continue using closed source debug components that were becoming difficult to enhance and maintain so we needed something new. We looked at existing open source debug technologies but concluded they wouldn't meet our needs. With the rare opportunity to start from scratch we decided we wanted a debugger that was perfectly tuned with Eclipse, focused on C/C++ development, had ground-up support for multiple cores/contexts, and a modern accessible design that would easily support new features. We also wanted to do the development in the open, in the same community that has supported our efforts over the past five years.

Building EDC has been much easier than it would have been a few years ago. The advances in the debug platform, the creation of the Debug Services Framework (DSF) in CDT, and the development of the Target Communications Framework (TCF) all provided essential building blocks. EDC is working on top of and between these components,  filling in the gaps and building out all the pieces required for a full working debugger.

While we wanted to develop EDC in the open we also wanted to get enough of the basics working so that when we contributed the code people would be able to easily try it out. So while we worked on all of the basic things a debugger should do we tried to keep the CDT community informed about our progress. Now that the starter code is in we'll be making our continuing work more visible, moving discussions into the eclipse bugzilla from our internal one. There is still a lot of work to do but now we're in a position to do it in collaboration with the community.

I've started building information about EDC on the CDT wiki here. EDC is not yet part of the regular CDT build but I hope to add it soon.

CDT-EDC Sprinting to 10-14

We've kicked off another sprint of EDC development: Warren is refactoring the Dwarf symbol reader to support an general symbol reader API that can be used for other symbol formats. Steve is adding support for more variable types: enums, arrays etc. Stephen is beefing up the Dwarf variable location to handle a lot of variable storage types we hadn't covered yet. Ling is cleaning up some bits of the Windows and Linux reference debuggers so they can reuse the existing CDT launch configurations. Chad is working on support for attaching to a running process. David is adding general support for consoles, integrating the Eclipse console view and data from the TCF streams service. Tim and Vasili are working on general stabilization and unit tests. I'm working on the beginnings of scripting support, outlining a DOM and prototyping how you can automate the EDC debugger from outside of Eclipse.

CDT-EDC Breakpoints

Finally a break in our long hot Austin summer with some rain and slightly cooler temps last weekend. In the eight hot weeks since my last post our team has been adding a bunch of new stuff to the EDC debugger we're preparing to contribute to CDT:

Symbian Device Connectivity - In addition to the our reference implementations on Windows and Linux we're now also debugging on a variety of Nokia phones. This has included supporting a new TCF debug agent and adding support for ARM to the existing x86 support in EDC. After using the existing TCF file system service to download everything we also created a new TCF service for installing applications on Symbian phones.

Breakpoints - Software breakpoints are now supported in a couple ways. If the TCF agent implements support for the TCF breakpoints service we'll use it, otherwise support is implemented in the DSF breakpoints service by writing memory to insert break instructions.

Disassembly - EDC now includes disassemblers for x86 and ARM that are built on a common disassembly framework. The disassemblers are used to populate the Disassembly View and to assist other operations like stepping.

Stepping - Stepping over/in/out of a range of code is now working both for TCF agents that know how to step over a range and those that can't. The Symbols service is used to determine the range of instructions to step over and then passes that to the agent. But if the agent doesn't support range stepping then the Run Control service will use the Disassembly service to find all the branch instructions and then set temporary breakpoints to perform the step.

One reason the breakpoint support was really interesting to implement was the interaction between several of EDC's DSF services. To support source level breakpoints you need to be able to get an address in some code from a source file it was built with and you need to know when and where the code is loaded. In EDC this begins when the debug agent sends a TCF context suspended event when a new code module is loaded. EDC's DSF Run Control service handles the event by sending a module loaded event to the other DSF services. The Module service adds a new entry to it's module list and then tells the Breakpoint service about the new module. The Breakpoint service asks the Symbol service if symbols are available for the new module. If so the Symbol service determines if the module was built using any of the source files that contain breakpoints. The Breakpoint service then asks the Symbol and Module services to provide memory addresses for the breakpoints, then installs them. Finally it tells the Run Control service to resume execution of the suspended context and your program begins running again. None of this is visible to the user except for a subtle change in the breakpoint marker to indicate it was installed successfully.

CDT-EDC Variables

It's over 100F/37C in Austin today and we've settled in for the lengthy sweltering drought they call "summer" here. It seems pretty quiet, partly because our Finnish colleagues are all on holiday and because our team has been decimated by some sort of stomach bug that has kept people home sick for several days at a time. Now that I've recovered a bit it's time for an EDC update.

Variables, at least the simple ones, are showing up now in the EDC debugger:

It all starts when one of the stack frames is asked for a list of its local variables. The EDC Symbol service supplies the list and the name of each variable is turned into an expression for the DSF Expression service to evaluate. When EDC evaluates the expression (which is really just the variable name) it needs to get the value of the variable.

For most simple types this is done by getting the location of the variable and reading memory from that address. Yet getting a variable's location out of the symbol information (DWARF 2 in this case) can be a complex multi-step process. For example, the symbol info might say integer X is stored 14 bytes off the stack pointer which is based on a register value with maybe an offset thrown in depending on where you are in the function. But eventually you get to read an value of the correct size (based on the variable's type) and you have something to use for the value of the expression, which is then dropped into the Variables View.

Of course we have a lot more to do with variables: complex types, formatting, location info etc. but the basic framework is in place to build on.

CDT-EDC Expression Parsing

I was fortunate to spend some time working at home recently with Max. He's a good coworker: sleeps much of the day and can be trusted with any confidential info that floats by. Sure, there were the occasional insistent interruptions, but no more than I get at the office.

Last sprint I started working on variable display in the EDC debugger and this quickly unraveled in a few interesting directions. One involves parsing and evaluating C/C++ expressions. The DSF framework considers all variables to be expressions, so if you want to display the value of variable "i" then you need to implement the same API used for more complex expressions like "i + 2" or "*((ShapePtr*)i).fore_color". I could probably work around this just to splash some simple variable values up but eventually we would need to parse and evaluate expressions anyway.

I approached this with some dread: the expression engine in our legacy debugger back-end was created in the mid 90's based on an early version of the CodeWarrior C++ compiler's front-end. As the C++ language evolved and the debugger was ported to support an endless series of now deceased architectures and platforms it's expression engine was extensively poked, prodded and patched. Eventually I would try to gather up all of the expression issues for a release and work on them all together. It usually took me a day of staring at the code to remember how it worked and some CVS sleuthing to discover why various "adjustments" were made. Then I would attempt to fix everything I could without breaking anything. Afterward I would go work on something else and forget the whole experience until the next release.

But when I first talked to Doug Schaefer about developing a new debugger in CDT he wondered if the CDT language parser could be used by the debugger for expression parsing. The debugger in the JDT does this, it uses the Java language parser to break an expression into actionable opcodes and then has an evaluation engine that executes those to get a value.

I set out to try this in the CDT and so far it is working great! After some startup pointers from Ed, and Mike, and Markus I've got the CDT language parser chewing on various expressions I give it and then producing a stack full of actions (add, subtract, dereference, look up value etc.) that get executed to produce a value. This can be used a lot of places in a debugger: in the Expressions View, when you hover in the editor, and in the Variable View's details pane. There is a lot more to flesh out but it's nice to be able to leverage so much existing work in CDT and build something that will be much easier to maintain in the future.

CDT-EDC Source Mapping

One of the basic things a debugger needs to do is show you where you are in your source code when you stop at a breakpoint or exception or after stepping. When this happens an EDC debugger's TCF debug agent sends it an event that includes a memory address for the location. If there is no other information available you'll end up in the Debug View with a stack frame location that looks like this:


The next step is to use the EDC-DSF Modules service to determine if this address is in some code the debugger knows about. The Modules service keeps track of all the bits of code loaded in the target process. If you can find a module for the address then you can show something more interesting like this:


Finally the EDC debugger checks to see if debug symbols are available for that module. Debug symbols include a mapping table of address ranges to line numbers in your source code. The EDC-DSF Symbols service checks to see if this information is available. If it can find a source location for the address then you can show a complete description like this:



So that's what I got working last sprint using the Windows debugger and an executable built with gcc/mingw and Dwarf2 for testing. After you have a source file to work with the existing source mapping facilities in CDT can handle finding missing source files and remapping files to new locations. Our legacy debugger back-end does it's own source mapping and it was a problem to keep it in sync with the mapping we are already doing in CDT. The whole process is simple faster, and less error prone in the EDC debugger.