Nokia Carbide.c++ on Eclipse

Update on Carbide, EDC, Symbian, Nokia

2011-05-16T13:01:00.000-06:00

Time for an update on the state of Carbide, the Eclipse Debugger for C/C++ (EDC), Symbian, and Nokia. As usual all comments and analysis are my own.

We finally shipped Carbide 3.2 to the public! This is the first publicly available 3.x version of Carbide and the first new version seen outside of Nokia in over a year. It includes support for a new build system, a brand new on-device debugger built on top of EDC, and lots of usability improvements. The delay was a result of one of the many iterations of Nokia's strategy over the past few years: for much of 2010 and until February 11th of 2011 we were telling developers that our mobile platform was the Qt app framework, not Symbian. The Qt folks were busy building their own IDE (Qt Creator) and so it was thought that developers would be confused if they had Carbide available as well. So Carbide was designated a "platform tool" only for use by Nokia's thousands of internal developers.

Then on February 11th our new CEO announced that Nokia would be using Microsoft's Windows Phone 7 software in the future and Symbian would be phased out over the next couple years. So our team has ended up in a strange position: we need to keep Carbide up to a high standard to support Nokia's continuing Symbian efforts but we know the plug will be pulled at some point, probably within the next year. Nokia has built in some flexibility for it's Symbian roadmap and arranged a soft landing by working out a deal to transfer 3000 people to Accenture at the end of this year. Accenture will then contract us back to Nokia to do the remaining Symbian work.

Our team's chief contribution to the Eclipse community over the past 24 months has been the Eclipse Debugger for C/C++ (EDC), part of the CDT project. Faced with the need for a new C/C++ debugger for Carbide and unhappy with any of the available options we designed and built one from the ground up. But the ground was already pretty rich: we leveraged all the work done in CDT's Debug Services Framework (DSF), used the CDT language parser for expression support, and relied on the Target Communications Framework (TCF) for low level debug services.

The first version of EDC shipped with Helios but it was pretty raw as we hadn't yet shipped a Carbide product using it. Indigo EDC is a big improvement in stability, performance, and feature set and we got to fix a number of things we didn't get right on the first attempt. The two reference debuggers provided with EDC (Windows & Linux) still aren't near production quality but are great for introduction and testing and provide a good example of how to build a debugger around EDC.

As we developed EDC we found other tools people interested in the idea and it reflects valuable contributions from the CDT community. As Carbide winds down I hope it continues to offer some value to anyone moving beyond their legacy debugger. I'd like to continue improving it and the rest of CDT but that awaits a volatile future.

Copying Memory

2011-03-17T16:04:00.002-06:00

I've been starting to clear some usability issues out of our backlog and ran into one that seemed simple enough but eventually required a little more thought and attention.

When debugging native code it's common to want to inspect memory so the Eclipse debug platform offers a standard Memory view. The view can host a number of memory renderings, each offering a different way of viewing a chunk of memory.

The "traditional" memory rendering is popular with C/C++ developers because it works much like the memory views in other debuggers and shows a range of memory in both binary and text formats.

"Traditional" memory rendering from CDT

Memory View in Xcode 4

Memory view from Visual Studio 2008

Memory window from CodeWarrior in 2001

Our developer (I'll call him Tom because that's his real name) wanted to do something that seemed pretty simple: select a hex value, copy it to the clipboard, and then paste it into a variable in the Variables view. But the global "Copy" command wasn't hooked up to the memory rendering so it did nothing. Tom then discovered "Copy to Clipboard" in the context menu.

But the "Copy to Clipboard" command copied all of the selected data from the rendering: addresses, binary, and text to the clipboard so it looked like this:

0x7FFF5FBFED20 74736574 00676E69 996E019D 3CBE8497 5FBFED40 testing...n...¾<@í¿_

Tom had to paste this into a text editor, then select just the binary/hex part, and finally had something to paste to set the value of his variable. This sort of thing delights some people because they find a work-around and then have another story to inflict on their colleagues, but Tom had real work to do, expected better, and logged a bug. Our team worked on other stuff that seemed more important until last week when I went back to take another look.

First I looked to see if the rendering's support for "Copy" could just do the right thing by grabbing the address, the binary data, or the text based on which part had been clicked in last. While that would work in the narrow case it wouldn't leave any indication of what it might do if you have forgotten which area you clicked in last, and the ability to copy all three (addresses, binary, text) would have been lost.

There needed to be a more explicit way to say which portion of the data you wanted to copy. So I changed the context menu to provide all four options.

This would make life better for Tom but still wouldn't fix the "select the hex data and hit control/command - c" scenario. So it also keeps track of one choice as the default option that is used when fired by the global "Copy" command, which is now hooked up to the rendering. Tom will have to select "Binary" once and then Copy will work the way he expected.

CDT-EDC Multi Context Debugging

2010-11-11T22:42:00.000-06:00

If you are ever at a gathering of the Eclipse CDT community and the conversation starts to drag just casually ask about support for debugging multiple processes, cores, or contexts at the same time. You'll hear about a lot of important interrelated issues and that the committers have been chipping away at them for some time.

One of the nice things about creating the EDC C/C++ debugger in CDT was the opportunity to include ground up support for debugging many contexts at once. But while that's fine under the hood we immediately ran into an important usability issue in the Debug view.

When you launch a debug session you start with a launch configuration that tells the debugger what to do and you end up with a Launch in the Debug View that shows the result. The Launch is usually named after the launch configuration that created it:

When you want to debug something else you can use another launch configuration that will create another Launch in the Debug View:

People find this confusing if both the things you are debugging are running in the same target environment. They expect them to be grouped together in the Debug view and they expect the Launch at the root of the view to represent the overall environment (phone, PC, multi-core system etc.)

The EDC debugger deals with this by letting new launches join existing ones if they are part of the same target environment. For example if you debug a Windows app (Simple.exe) you will get a new debug session and Launch in the Debug View.

Debug a second Windows app (BlackFlag.exe) and it will join the existing one:

EDC uses a description and image of the target environment provided by the debugger implementation. In that case it was the Windows machine name. In this one it's the name of my phone:

Of course there are lots of other issues related to multi context debugging. To see how the CDT community is addressing these you can follow along with the CDT multi-core working group.

CDT-EDC Where is the Exit?

2010-09-07T15:45:00.001-06:00

Our team is working on our next generation of C/C++ development tools for Symbian, Carbide 3.0 (C3). These build on the Eclipse Debugger for C/C++ (EDC) that's part of the CDT project. Our test team reported a bug that raises an interesting question for developers: do you want the debugger to show you what's really happening even if it seems a bit confusing?

Here is the situation: you are debugging a C++ function that looks something like this:

Say you are stepping through this function in the debugger and you start at line 31. "value" happens to be 200 so the condition on line 31 fails. You step over and end up on line 36. Step over again to go to line 37. So far everything looks OK. Now you step again from line 37 and you might reasonably expect to exit the function. Instead you end up on line 34! This looks like a heinous bug but is a case of the compiler reorganizing things even in a normal debug build with optimizations off.

The compiler has built this function by putting all of the return/exit code in one place, in the middle of the function at line 34. At the end of the function, at line 37, instead of generating the exit stuff again it simply jumps to the address at line 34 and exits there. It dutifully and accurately records this bit of slight-of-hand into the symbol information and the EDC debugger shows you on line 34.

Should the EDC debugger try to detect this just to avoid confusion? Another perspective is that the debugger should show you exactly what's happening even if the result is sometimes puzzling. What do you think the EDC debugger should do?

Giant Symbols

2010-06-17T22:16:00.000-06:00

The Carbide team here at Nokia has spent the last couple months working on getting the Carbide 3.0 (rebranded C3) beta out to developers, doing an update to the current tools (Carbide 2.6) and wrapping up our contributions to the Eclipse Helios release, mostly in the C/C++ Tools project (CDT).

In the midst of all that I was asked to help a developer who is porting a very large productivity app to Symbian. His problem involved using the debugger in Carbide 2.x and the size of the symbol files generated for his application. A symbol file is generated by the build tools and contains the information used by the debugger to make sense of your source code, variables, and most of the other stuff you see in a debug session.

With all the apps developed at Nokia we've seen a lot of large symbol files but this one was nearly a 1GB, almost double the size of our largest sample. Our current tools (Carbide 2.x) crashed when trying to open the big symbol file and couldn't easily be fixed to handle something that size.

This is one of the use cases we had in mind when we developed the new debugger in C3, the Eclipse Debugger for C/C++ (EDC), part of the Eclipse CDT project. So we ran our new big symbol file sample through the EDC symbol reader unit tests and everything worked quickly and smoothly. I gave the developer a beta build of C3 and once we worked through some setup issues on Windows 7 he was able to debug the app with the giant symbol file and see all the debug info.

Then he says "that's the symbol file for just one of our dlls, all together we actually have about 14GB of symbol data." So I think we have now put the EDC debugger in C3 through a least one kind of worst case scenario.

While all this was going on someone asked why the symbol files were so large to begin with. This big productivity app has been ported to lots of platforms with different tool chains and they never ended up creating such a large amount of symbol data. After some investigation they discovered a build flag called "remove_unneeded_entries" that, when enabled, dramatically reduces the size of the symbol data. We still have more testing to do but so far it looks like it really helps debugger performance with no loss of useful data.

Carbide, CDT & EDC at EclipseCon 2010

2010-03-30T16:24:00.000-06:00

Had a great time at EclipseCon in Santa Clara last week. Some highlights:

Face time with our colleagues from Nokia, the Symbian Foundation, Eclipse and CDT communities. Ronnie King, Warren Paul, and I came from the Austin office and had a great time catching up with people we usually don't get to see in person very often. I missed EclipseCon last year but I heard attendance was up and organizations were able to send more people this time.

I really liked the session format: the reduced times seemed to really focus the presentations and kept a wide variety of material flowing throughout the day. I know the CDT session Doug and I did seemed to fly by quickly.

Had a great response to the new EDC (Eclipse Debugger for C/C++) that will be in Carbide 3.0 and our team is contributing to CDT 7.0. People liked the overall concept, features, and extensibility and I think we'll soon start seeing more contributions from the community.

Lots of discussions about usability and performance, both very important to us for Carbide. There is new work going on in the Eclipse & CDT communities that will help us smooth out some rough spots our Carbide users have had to put up with.

Visiting developers at the Nokia office in Mt. View. We love getting to shadow developers around and watch them work. Great unfiltered feedback. We had a chance to introduce them to some features in Carbide they didn't know about and help them through some difficult issues that are tough to understand unless you can look over the person's shoulder.

Aligning Carbide 3.0 with Eclipse Helios

2010-02-24T10:44:00.000-06:00

Like most teams that build C/C++ tools on top of Eclipse & CDT we have been building Carbide with the latest platform release and a somewhat forked version of CDT. This made sense as most of our releases have shipped in the winter or spring, long after the last major Eclipse release. But with the next major release of Carbide due after Helios we’ve jumped on the opportunity to align more closely on Eclipse 3.6 and CDT 7.0. How closely? Well, we’ve just started creating our Carbide builds with 3.6M5 and the CDT 7.0 sources straight out of the eclipse.org CVS repository.

We hesitated to do this in the past because it means we effectively have the entire CDT team contributing to Carbide in real time: any one of the committers can break our build or introduce bugs. But we’re more confident now and our experience with the CDT community has been very good: people rarely break the build and the quality and review of contributions has been excellent.

And since we sync up with the latest platform and CDT eventually anyway is makes more sense to discover any issues at the moment when the work is developed and contributed, not months later. Now both our internal test team and the Carbide beta group can give the new platform and CDT stuff a workout before it’s released. This also lets us get rid of our internal copy of CDT we would occasionally sync with the public one. Trying to keep it updated was a pain and it created a temping way for us to make changes that weren’t really ready to go back into the community.

CDT-EDC and Openness

2010-02-05T11:12:00.000-06:00

While our team here at Nokia has been developing the Eclipse Debugger for C/C++ (EDC) as a component of the CDT project we wanted to do it in the open as much as possible. But we still had quite a bit of initial work to do before the community would be able to evaluate what we were doing. Now that the initial contribution is in CDT we're going to start working on changing our habits to bring more openness and transparency to the effort:

We're going to start moving our discussions about EDC to the cdt-dev mailing list. This may seem simple but we also don't want to bore the CDT community with issues related to Carbide, Symbian, on-device debugging etc. so we'll need to be a little selective.

New issues with EDC will be logged in the Eclipse bugzilla instead of our Carbide bugzilla. We'll need to be a little selective here too and probably won't log every minor issue, but anything that merits any sort of discussion will be public.

Currently our team has two committers on the CDT project, Warren Paul and myself. I've been syncing the EDC sources in CDT with our internal copy. This has done wonders for my committer stats, it looks like I wrote all of the EDC code, but has masked the contributions of our team members. So we're going to start nominating additional people as CDT committers, starting with the ones who already have a track record of contributions to other components in addition to their work on EDC.

Carbide, CDT, EDC Update

2010-01-28T16:51:00.000-06:00

Time for an update from our team here at Nokia working on Carbide 3.0, Eclipse CDT 7.0, and the new Eclipse Debugger for C/C++ (EDC):

EDC is now part of the regular CDT build. Many thanks to Doug Schaefer for getting this done. I hadn't done much with the CDT build before and wasn't making much progress, but Doug resolved a couple tricky issues, like pulling the TCF source in from Subversion. We still have a version of EDC in our CVS so the Carbide team can easily commit stuff, but I'm syncing it every few days.

Custom Variable Formatting in EDC: David committed the first phase of support for custom variable formatting. Using a new extension point a plug-in can alter the way data is shown in the Variables view during an EDC debug session. An extension can provide an alternative to the standard structure and content of any variable type.

We intend to provide formatters for some STL, Qt & Symbian data types. Also there are general purpose formatters for arrays and structs:

Unformatted C/C++ struct

Formatted C/C++ struct

Debug Snapshot Albums: A new feature in EDC is the ability to capture a snapshot of the state of your debug session for later review. A new Snapshot Album view shows the available albums of snapshots from prior debug sessions and lets you annotate their contents. Snapshot Albums can be launched to show a re-creation of the original debug session. Tim and I have been overhauling some of the initial work and getting the UI far enough along so that we can start usability testing soon. I'll write more about this when the work settles down a bit.

Plug & Play Debug Connectivity: We've made a lot of improvements in usability when starting a debug session on a phone. We started with a detailed analysis of all the options available when connecting to a phone, then narrowed them down into a new launch wizard that can start a debug session in just a few steps. This will make on-device debugging in Carbide a lot easier and I hope to apply some of the lessons learned when we overhaul the launch experience in CDT.

EDC Symbol Reader Overhaul: Ed & Steve are in the midst of an overhaul of the Dwarf reader in EDC. We originally wrote it for completeness and to have something to build the rest of the debugger around, but knew more work was required to get better performance and memory usage. This effort will likely be continuing for the next several weeks.

Community Contributions to EDC: People are starting to review the code, report bugs, and contribute patches. In particular John Cortell at Freescale has been improving the way EDC connects to TCF agents and providing a fresh perspective on everything from naming to API visibility.

That's it for now, I'm diving back into the code...

Eclipse Debugger for C/C++ (EDC)

2009-12-02T11:49:00.000-06:00

I'm very happy to announce that the sources for EDC, the Eclipse Debugger for C/C++, are now in the Eclipse CVS repository. EDC is a new component of the CDT project that provides a complete architecture for C/C++ debugging within Eclipse/CDT.

The Carbide team started developing EDC after setting on a new course that would open up nearly all of our development tools to either the Eclipse or Symbian communities. We couldn't continue using closed source debug components that were becoming difficult to enhance and maintain so we needed something new. We looked at existing open source debug technologies but concluded they wouldn't meet our needs. With the rare opportunity to start from scratch we decided we wanted a debugger that was perfectly tuned with Eclipse, focused on C/C++ development, had ground-up support for multiple cores/contexts, and a modern accessible design that would easily support new features. We also wanted to do the development in the open, in the same community that has supported our efforts over the past five years.

Building EDC has been much easier than it would have been a few years ago. The advances in the debug platform, the creation of the Debug Services Framework (DSF) in CDT, and the development of the Target Communications Framework (TCF) all provided essential building blocks. EDC is working on top of and between these components, filling in the gaps and building out all the pieces required for a full working debugger.

While we wanted to develop EDC in the open we also wanted to get enough of the basics working so that when we contributed the code people would be able to easily try it out. So while we worked on all of the basic things a debugger should do we tried to keep the CDT community informed about our progress. Now that the starter code is in we'll be making our continuing work more visible, moving discussions into the eclipse bugzilla from our internal one. There is still a lot of work to do but now we're in a position to do it in collaboration with the community.

I've started building information about EDC on the CDT wiki here. EDC is not yet part of the regular CDT build but I hope to add it soon.

CDT-EDC Sprinting to 10-14

2009-10-05T13:20:00.000-06:00

We've kicked off another sprint of EDC development: Warren is refactoring the Dwarf symbol reader to support an general symbol reader API that can be used for other symbol formats. Steve is adding support for more variable types: enums, arrays etc. Stephen is beefing up the Dwarf variable location to handle a lot of variable storage types we hadn't covered yet. Ling is cleaning up some bits of the Windows and Linux reference debuggers so they can reuse the existing CDT launch configurations. Chad is working on support for attaching to a running process. David is adding general support for consoles, integrating the Eclipse console view and data from the TCF streams service. Tim and Vasili are working on general stabilization and unit tests. I'm working on the beginnings of scripting support, outlining a DOM and prototyping how you can automate the EDC debugger from outside of Eclipse.

CDT-EDC Breakpoints

2009-09-15T13:02:00.014-06:00

Finally a break in our long hot Austin summer with some rain and slightly cooler temps last weekend. In the eight hot weeks since my last post our team has been adding a bunch of new stuff to the EDC debugger we're preparing to contribute to CDT:

Symbian Device Connectivity - In addition to the our reference implementations on Windows and Linux we're now also debugging on a variety of Nokia phones. This has included supporting a new TCF debug agent and adding support for ARM to the existing x86 support in EDC. After using the existing TCF file system service to download everything we also created a new TCF service for installing applications on Symbian phones.

Breakpoints - Software breakpoints are now supported in a couple ways. If the TCF agent implements support for the TCF breakpoints service we'll use it, otherwise support is implemented in the DSF breakpoints service by writing memory to insert break instructions.

Disassembly - EDC now includes disassemblers for x86 and ARM that are built on a common disassembly framework. The disassemblers are used to populate the Disassembly View and to assist other operations like stepping.

Stepping - Stepping over/in/out of a range of code is now working both for TCF agents that know how to step over a range and those that can't. The Symbols service is used to determine the range of instructions to step over and then passes that to the agent. But if the agent doesn't support range stepping then the Run Control service will use the Disassembly service to find all the branch instructions and then set temporary breakpoints to perform the step.

One reason the breakpoint support was really interesting to implement was the interaction between several of EDC's DSF services. To support source level breakpoints you need to be able to get an address in some code from a source file it was built with and you need to know when and where the code is loaded. In EDC this begins when the debug agent sends a TCF context suspended event when a new code module is loaded. EDC's DSF Run Control service handles the event by sending a module loaded event to the other DSF services. The Module service adds a new entry to it's module list and then tells the Breakpoint service about the new module. The Breakpoint service asks the Symbol service if symbols are available for the new module. If so the Symbol service determines if the module was built using any of the source files that contain breakpoints. The Breakpoint service then asks the Symbol and Module services to provide memory addresses for the breakpoints, then installs them. Finally it tells the Run Control service to resume execution of the suspended context and your program begins running again. None of this is visible to the user except for a subtle change in the breakpoint marker to indicate it was installed successfully.

CDT-EDC Variables

2009-07-15T10:13:00.009-06:00

It's over 100F/37C in Austin today and we've settled in for the lengthy sweltering drought they call "summer" here. It seems pretty quiet, partly because our Finnish colleagues are all on holiday and because our team has been decimated by some sort of stomach bug that has kept people home sick for several days at a time. Now that I've recovered a bit it's time for an EDC update.

Variables, at least the simple ones, are showing up now in the EDC debugger:

It all starts when one of the stack frames is asked for a list of its local variables. The EDC Symbol service supplies the list and the name of each variable is turned into an expression for the DSF Expression service to evaluate. When EDC evaluates the expression (which is really just the variable name) it needs to get the value of the variable.

For most simple types this is done by getting the location of the variable and reading memory from that address. Yet getting a variable's location out of the symbol information (DWARF 2 in this case) can be a complex multi-step process. For example, the symbol info might say integer X is stored 14 bytes off the stack pointer which is based on a register value with maybe an offset thrown in depending on where you are in the function. But eventually you get to read an value of the correct size (based on the variable's type) and you have something to use for the value of the expression, which is then dropped into the Variables View.

Of course we have a lot more to do with variables: complex types, formatting, location info etc. but the basic framework is in place to build on.

CDT-EDC Expression Parsing

2009-06-18T12:59:00.014-06:00

I was fortunate to spend some time working at home recently with Max. He's a good coworker: sleeps much of the day and can be trusted with any confidential info that floats by. Sure, there were the occasional insistent interruptions, but no more than I get at the office.

Last sprint I started working on variable display in the EDC debugger and this quickly unraveled in a few interesting directions. One involves parsing and evaluating C/C++ expressions. The DSF framework considers all variables to be expressions, so if you want to display the value of variable "i" then you need to implement the same API used for more complex expressions like "i + 2" or "*((ShapePtr*)i).fore_color". I could probably work around this just to splash some simple variable values up but eventually we would need to parse and evaluate expressions anyway.

I approached this with some dread: the expression engine in our legacy debugger back-end was created in the mid 90's based on an early version of the CodeWarrior C++ compiler's front-end. As the C++ language evolved and the debugger was ported to support an endless series of now deceased architectures and platforms it's expression engine was extensively poked, prodded and patched. Eventually I would try to gather up all of the expression issues for a release and work on them all together. It usually took me a day of staring at the code to remember how it worked and some CVS sleuthing to discover why various "adjustments" were made. Then I would attempt to fix everything I could without breaking anything. Afterward I would go work on something else and forget the whole experience until the next release.

But when I first talked to Doug Schaefer about developing a new debugger in CDT he wondered if the CDT language parser could be used by the debugger for expression parsing. The debugger in the JDT does this, it uses the Java language parser to break an expression into actionable opcodes and then has an evaluation engine that executes those to get a value.

I set out to try this in the CDT and so far it is working great! After some startup pointers from Ed, and Mike, and Markus I've got the CDT language parser chewing on various expressions I give it and then producing a stack full of actions (add, subtract, dereference, look up value etc.) that get executed to produce a value. This can be used a lot of places in a debugger: in the Expressions View, when you hover in the editor, and in the Variable View's details pane. There is a lot more to flesh out but it's nice to be able to leverage so much existing work in CDT and build something that will be much easier to maintain in the future.

CDT-EDC Source Mapping

2009-06-01T15:00:00.001-06:00

One of the basic things a debugger needs to do is show you where you are in your source code when you stop at a breakpoint or exception or after stepping. When this happens an EDC debugger's TCF debug agent sends it an event that includes a memory address for the location. If there is no other information available you'll end up in the Debug View with a stack frame location that looks like this:

The next step is to use the EDC-DSF Modules service to determine if this address is in some code the debugger knows about. The Modules service keeps track of all the bits of code loaded in the target process. If you can find a module for the address then you can show something more interesting like this:

Finally the EDC debugger checks to see if debug symbols are available for that module. Debug symbols include a mapping table of address ranges to line numbers in your source code. The EDC-DSF Symbols service checks to see if this information is available. If it can find a source location for the address then you can show a complete description like this:

So that's what I got working last sprint using the Windows debugger and an executable built with gcc/mingw and Dwarf2 for testing. After you have a source file to work with the existing source mapping facilities in CDT can handle finding missing source files and remapping files to new locations. Our legacy debugger back-end does it's own source mapping and it was a problem to keep it in sync with the mapping we are already doing in CDT. The whole process is simple faster, and less error prone in the EDC debugger.

CDT-EDC Launching

2009-05-18T14:00:00.001-06:00

The team mostly worked on launching EDC debug sessions last sprint. We polished up the Windows launch so terminate shuts everything down correctly and modified the debug agent to shut itself down if no one was connected to it for a short time There are several launch types in Carbide for Symbian debugging that we thouht we might have to clone for EDC. But we've given them the same treatment as Pawel gave the launches in CDT 6.0: defining one standard set of launch types but assigning multiple launch delegates to each one. I wasn't aware you could do this until Pawel implemented this for the CDT launches. It's great for Carbide because we can reuse our existing launch support and just implement an EDC launch delegate. We probably wouldn't ship Carbide with both CDI and EDC launches enabled for the same type, but it will be great for testing and development.

Traditionally most C/C++ launches synchronously start a process and then debugging begins. The simple EDC launches start a process asynchronously and then wait for a process created event from the debug agent. Later we'll support more complex launches: ones that debug multiple processes, target specific DLLs and wait for a process to load them, or simply let you look at stuff on the target platform before debugging anything.

CDT-EDC Baby Steps

2009-05-04T11:20:00.000-06:00

Our team runs two week sprints of development activity in pretty standard Agile & Scrum fashion. That seems like a good interval to post updates on our work on the Eclipse Debugger for C/C++ (EDC).

While our intention is to develop EDC as an integral part of the CDT project we haven't committed any of it yet: we want to get more stuff working and the community is too busy getting CDT 6.0 ready to look at it much anyway until after Galileo ships.

I spent much of the last sprint doing bits of high level design work but got back into the code to finish up a story Warren Paul was working on until he left for the hospital.

There has a been a baby boomlet in the CDT committer community over the past several months. Our baby Max arrived back in February. Then Pawel Piech had a new one arrive about a month ago. And last sprint it was Warren's turn.

So I finished up work he was doing on how the EDC debugger connects to TCF agents. We wanted the clients of the agents, usually DSF services, to not know much about TCF concepts like Peers or Channels, but to just be able to ask for "a run control service for Windows debugging." We created a TCFServiceManager and an TCFAgentDescriptor extension point to hide all the details of launching and getting a TCF service from an agent. The extension contains a property list that describes the agent and a delegate that knows how to start it up. The service manager finds the descriptor that matchs the client's request, asks the delegate to launch the agent if it isn't already in the TCF Peer list, then opens a Channel and returns the requested service. The agents get to decide when to shut themselves down, usually when no one has been connected to them for a while.

When an EDC debug session starts we hook up the DSF services to whatever TCF services they need to use. So now the EDC Windows debug agent is started automatically when you start a debug session and shuts down shortly after the session terminates.

The EDC Windows TCF agent is written in C/C++ but we also wanted a reference TCF agent written in Java that can be run in Eclipse or in another instance of the VM. Ling Wang produced one in a previous sprint and just finished adapting it to talk to GDB Server for use by the EDC linux debugger. We'll do something similar for Symbian OS device debugging except it will talk to TRK, the debug agent that runs on the phone.

Each sprint we try to have a simple ramp-up story we can give to another member of the larger Carbide team to help bring them up to speed on EDC. This time it was adding support for writing register values to the EDC Windows debugger (reading them already worked). Stephen Chong came in cold having never seen DSF or TCF but knocked it out in a few days, complete with unit tests.

Instant API

2009-04-09T12:23:00.005-06:00

One of the aspects of our current financial crisis I've found interesting is the ability of the US Federal Reserve bank to create additional money out of thin air. How does that work? Do they have some money supply control site where Ben Bernanke enters a number and hits the "Submit" button?

I was reminded of this during this month's CDT call when we discussed fixing some bugs that require making a few non breaking API changes. This is an issue because it's after the API freeze date but I think our underlying problem is how through the best intentions we've created a bunch of API from code that wasn't intended to be.

When I started working on CDT the definition of API was a little loose but mostly described interfaces in a public package that we designated API in the CDT docs. Over time we moved to tighten this up and then finally decided to use the same definition as the platform: roughly any interface or class in a non-internal package was to be considered API. Having the API tooling helped with this considerably and newer code like DSF that was designed with this in mind moved into CDT to set a good example.

But this also means that lots of code in CDT that was probably never intended to be API instantly became API. Sure, it probably could have been designed better in the first place or we could have moved it to internal packages or I could have thought about this more at the CDT Summit instead of watching for the afternoon cookies to be delivered. But now here we are needing to fix bugs and needing to make non breaking changes to do it. On the call I said I thought we should just go ahead and make the changes we need and work on a better plan for this next time.

We've just been through something like this with our Carbide sources. We had put all the stuff we thought of as public API into a special SDK plug-in. But as part of open sourcing Carbide for the Symbian Foundation we found ourselves needing to conform to the Eclipse platform definition of API. So we kept ourselves busy for several weeks refactoring things into internal packages so we wouldn't suddenly make a bunch of stuff API that was intended to be internal and private. Now that we've finished and we're testing the result we're finding that we've broken plug-ins people have written for Carbide that were using "non-API" stuff that we renamed. We're reevaluated what we need to make real published API now but it's a good lesson that people will do what they need to in order to get their plug-ins to work and worry about the consequences later on. After all no one wants to create a tool and say to their customers "It doesn't solve your problem or work the way you would expect, but we only used the published APIs!" I guess it's a convoluted form of API requirements gathering.

Logging & Tracing

2009-03-25T15:00:00.002-06:00

Since I'm not at EclipseCon this week I trying to get some real work done here in Austin now that the crowds from SXSW have left town. I've been looking into the best way to build good diagnostic logging into the EDC debugger, mostly as a support tool. When people report problems that we can't reproduce we can ask them to create a log of debugger activity that can help us pinpoint the problem.

In Carbide we have a simple diagnostic log based on java.util.logging that works pretty well but then I noticed that Eclipse 3.5M6 includes a new Equinox tracing API. When your application normally keeps a log of activity it is probably best to use one of the various logging services to help manage it. But in this case I just want to be able to get more information to help debug problem situations. It looks like the new tracing API will be a nice way to do this: there is a simple way to trace function entry, exit, exceptions, and simple messages. Each trace call can be tied to a particular debug option so we should be able to turn on tracing for specific parts of the code without getting details about everything at once.

For example this traceEntry call includes the debug option for the EDC run control service and will log the properties passed in and on exit will log the new object created.

public ExecutionDMC contextAdded(Map properties) {
EDCDebugger.getDefault().getTrace().traceEntry(RUN_CONTROL_TRACE,properties);;
ThreadExecutionDMC newDMC = new ThreadExecutionDMC(this, properties);
getSession().dispatchEvent(new StartedEvent(newDMC), RunControl.this.getProperties());
EDCDebugger.getDefault().getTrace().traceExit(RUN_CONTROL_TRACE, newDMC);
return newDMC;
}

Eventually we'll need some UI to help people turn this on and off and maybe some tools to help them send the file or attach it to a bug.

Eclipse Debugger for C/C++ (EDC)

2009-03-12T15:40:00.004-06:00

In my last post I talked about how the Eclipse C++ debugger landscape had changed and led us to want a new debugger that’s perfectly tuned to work with CDT/DSF and Eclipse. The existing work in the debug platform and CDT/DSF provides a nice set of services that include user interface elements, commands, and APIs for debug data access. What’s missing is the part of the debugger that knows how to interact with the thing you are debugging to actually get the memory and register values, do run control, provide variable values etc. Traditionally this has been provided by a debugger back-end of some sort: CDT includes an integration of DSF and gdb while other tools vendors have a similar debugger engine, often extracted from their legacy product.

When taking a fresh look at this it didn’t make much sense to create a monolithic debugger back-end and try to integrate it with CDT/DSF. DSF already provides a well designed service model, what’s missing is a complete set of services to flesh out a debugger. Extending the reach of the DSF services into things traditionally done by a debugger back-end would also let us design and develop most of it using the same language, tools, and environment as all of our other Eclipse work.

Yet some services still need to reach out to native code either to interact with a native debug API or to do something performance intensive like parsing symbol files. So we would need a way for our DSF services to talk to the native code that is likely running in a different process or on another device. Fortunately the DSDP/TM/TCF project solves exactly this problem. TCF is a protocol that lets an agent (usually in native code) provide a discoverable set of services to clients (in this case our DSF services). This lets our new DSF services do as much of the work as they can and then use the TCF protocol to talk to remote devices or native code as required.

This collection of DSF services and TCF agents will form the foundation for a complete C/C++ debugger in Eclipse. So we're calling this project the Eclipse Debugger for C/C++ (EDC).

Building a new solution that will let us provide powerful, stable, maintainable, and consistent debugging for the platforms we support (Symbian, Windows, Maemo) means developing DSF services and TCF agents that target them. We’ll end up with a core set of common EDC plug-ins and agents and others that add specific support for a device or operating system. Anything of general interest will be developed within the CDT community and we'll put the Symbian specific work in the Symbian Foundation.

I'll post more details as we develop stuff.

C/C++ Debugger Retrospective

2009-03-12T15:30:00.001-06:00

We're doing some new work on the debugger in Carbide and for some perspective I'll take a look back at how the Eclipse C/C++ device debug landscape has changed in the past few years.

When we first began designing our Carbide.c++ tools for Symbian mobile development a few years ago there was a lot of new activity around C/C++ debugging in the Eclipse community: The CDT project offered a proven set of APIs for debugger integration (CDI), but as vendors continued to move their existing debuggers into new Eclipse based environments it was clear that more work needed to be done to meet the expectations of veteran C/C++ developers.

Our team began creating C/C++ tools for Symbian using the CodeWarrior IDE and when we designed a new generation of Eclipse based tools we decided to reuse its proven debugger engine and hook it up to CDT/CDI. While this let us quickly deploy a good C/C++ debugger as an initial step, we also carefully watched developments in the device debugging project (DSDP/DD) and the creation of the Debug Services Framework (DSF, now part of CDT). It was also exciting to see the many improvements resulting from the debug platform team’s engagement with the device debug community. We began developing a list of improvements we would like to make to Carbide's debugger that would be difficult to do without using the new work in the platform and DSF.

At one point I started prototyping an integration of our debugger engine and DSF. While I got a number of things working it also became apparent that it wasn’t a very good fit: DSF uses a highly asynchronous service model for supplying information about the debug session and our debug engine didn’t work that way. I was having to rig up a lot of stuff in a clumsy way to make it work with DSF. To move our debugger forward it was looking like we would not only need to move to DSF but also rework a lot of our debug engine.

Doing that work looked difficult because the engine code was pretty convoluted and much of it needed an overhaul. Also the engine was closed source licensed code and this would be a issue as we moved to open source all of Carbide along with the Symbian Foundation.

So that’s how we have moved from our initial legacy debugger integration with CDT/CDI to how it looks today: lots of promising development in the CDT and device debug community but we're not yet able to take advantage of much of it with our existing debugger. So we started thinking about something new that we can develop in the open and tune to work perfectly with DSF and Eclipse.

Not Going to EclipseCon

2009-03-02T16:58:00.003-06:00

After making the trip the past several years I'm going to have to miss EclipseCon this time. I'd like to be going: I'll miss the sessions, BOFs, and most of all the informal conversations with all the interesting people in the Eclipse community.

But this year our Baby Max arrived a bit early and so I'll be home helping with diapers etc. Maybe I can catch up with everything on Planet Eclipse when I'm up in the small hours.

Carbide 2.0 Update

2009-01-12T16:32:00.002-06:00

I've somehow let the past several months go by without posting anything so it's time to catch up. Our team launched Carbide 2.0 late last year. It's built on top of Eclipse 3.4 and CDT 5.0 with all of our tools for Symbian OS development built in. It was an unusual release to work on because a lot of the work went into supporting things that are not immediately apparent to someone moving up from Carbide 1,3, For example we added support for Qt on S60 but until October when the tech preview was available most people couldn't try it out. There is a new build system for Symbian that we now support but it isn't widely used yet. And as more teams at Nokia move their projects to Carbide we added support for stuff they use but isn't yet ready for the whole development community. Even many of the improvements in CDT 5.0 result in much better C++ code navigation, but are not big visible new features.

So some people were a little unimpressed at first ("why is this Carbide 2.0? It feels like 1.4") but later came back with much more positive feedback as they found the new work and improvements to existing features.

But one of the best changes came at the end when we decided that all editions of Carbide should be free to anyone. While we have always had a free edition it didn't include some higher end tools for debugging and analysis. This will help grow the Symbian developer community, and our team is thrilled that our code no longer has to do all that license checking.

System Search - Finding Files Anywhere

2008-06-30T15:51:00.005-06:00

Now that CDT 5.0 is out the door I'm starting to review some of the work the Carbide team has done over the past year and pick out things that might make good contributions to the next CDT release.

As we talk to C++ developers who have moved to our Eclipse based tools we occasionally ask them if they are working in Carbide all of the time or still need to their old IDE environment around. The old IDE is usually Visual Studio or CodeWarrior and we found that people were switching back so they could use them to search for and in a collection of files. C++ developers often need to work with files strewn all over the file system so the regular File Search in Eclipse, which only searches files in the workspace, doesn't do enough for them.

Of course there is a bug for this but we quickly came to the conclusion that we couldn't easily modify or extend standard File Search so we created a System Search that would let you look at files anywhere in the file system.

This should look pretty familiar except for the field at the bottom that lets you pick a location in the file system to search. The results are displayed in an adaptation of the standard search results view.

We've gotten a lot of very positive customer feedback but people have also been quick to ask for more features. There is also some more work we need to do before contributing this to CDT: a little more code cleanup is needed and more testing on multiple platforms.

If you want to play around with System Search the code is attached to the bug.

Austin Eclipse DemoCamp

2008-06-26T14:10:00.004-06:00

I'm back at my desk now trying to stay awake after a coma inducing lunch of Stubb's BBQ following the Austin DemoCamp. A bunch of Nokia people from our office joined Chris Aniszczyk and people from IBM, Freescale, and others to look at a few demos and talk about Ganymede. (That's Chris pictured along with John Cortell and Glenn Meter).

Despite developing Eclipse plug-ins all day I haven't spent much time looking at the new PDE stuff in Ganymede so it was great to talk to Chris about it first hand. I talked a bit about CDT and how we develop Carbide.

The session was also something of a reunion of former Metrowerks people: the team that developed the CodeWarrior tools was bought by Motorola years ago which then spun them out along with Freescale. The CodeWarrior for Symbian team then moved to Nokia and now develops Carbide.c++ at our office here next to the coma inducing BBQ. So a lot of people who created the CodeWarrior tools back then are working with Eclipse now. It was great to have a chance to visit with some old friends and make some new ones. I'm hoping we can continue to get the Eclipse people in Austin together now and again.

Annoucing Eclipse b4

2008-04-01T00:11:00.001-06:00

Details were unveiled today about project b4, a new release of Eclipse focusing on the past. Project lead Paul Rubens said "We're especially interested in the Baroque era, but all the contributions to Eclipse over the centuries will be gathered in this release."

An example of the historical breadth of the project can be seen in a patch contributed by Roman Centurion Lucius Vorenus. "We were a little concerned about support as the author has been dead for over 2000 years, but his patch applied cleanly so we decided to go for it." said Rubens. "The patch introduced the ISlaveResource interface and all of his unit tests passed."

Backward compatibility is an important goal of b4. "The ruins in Cyrenaica have yielded some interesting early API inscribed in stone tablets. We plan to support them all, except the ones that were already broken when we found them."

Researchers have also deciphered a patch hidden in the ceiling of the Sistine Chapel. "We normally would only take a patch if it is attached to a bugzilla, but the sheer beauty of this one, all in the original Latin, is too good to pass up. Besides it's just an NPE fix."

The contributions to b4 often reflect the hazards of a time when many committers' work was interrupted by violence and disease. "Many of the project teams were decimated by the Black Death, and a lot of promising contributions are unfinished. They won't compile because they end with phrases like 'castle breached, the end is near' or simply 'Aaaaarrrrrrggghhh'."

Development of b4 will be completely open and transparent according to Rubens: "Certainly it will be open, just as Roman citizenship was open to anyone who lived in the city." He hastened to add "But that was later extended to most of Italy and then even to some barbarian tribes."

The final release date for b4 is still uncertain "There are several patches in the Vatican archives we want, but now the Pope is involved in the IP review process and we don't know how long it will take."

Carbide 1.3 End Game

2008-02-25T11:15:00.000-06:00

We are in the last weeks of testing Carbide 1.3: the code is frozen, the test team is rechecking everything, and the beta testers are trying out the release candidate build. This is usually the time when some bug that has been around all along makes a surprise appearance in a really troubling way.

Last week we started getting some reports of the entire workbench locking up. People who were really beating on the tools all day said that often after a paste or save operation the UI would be unresponsive. Of course we couldn't reproduce the problem but we had two interesting clues: some people said the UI started responding again after a while and one beta tester sent in a screenshot of the frozen workbench.

Everyone has their own personal "time out" setting: the amount of time they are willing to wait for an unresponsive application to start working again before they kill it or restart their computer or chuck it over the cube wall. So that some people said the workbench came back to life after a while probably indicated we had some lengthly process blocking the UI thread, not that Carbide was frozen forever.

The screenshot gave us a few more clues: the progress indicator said the CDT indexer job was just starting and that the text the user was pasting hadn't yet appeared in the editor. I started trying to reproduce the problem by forcing a project to reindex and then repeatedly pasting code.

After a lot of pasting we found that the problem had nothing to do with the paste operation but with the use of the control key (as in ctrl-v to paste or ctrl-s to save). With the control key down CDT was asked to produce a hyperlink. Normally this happens quickly because the file has already been indexed and the results cached, but when it's dirty it proceeds to do some indexing on the UI thread which leaves the workbench completely unresponsive.

It seemed like the right thing to do in this case was to just not do the indexing if the cache was stale. We turned to the CDT team to confirm this and within a few hours we had changes in place to prevent this problem and a couple other related issues too.

So now I'm starting this week like I did the last: hoping Carbide 1.3 can slide out the door without any serious issues popping up.

Looking at DSF

2008-02-20T11:38:00.000-06:00

Our next major release of Carbide.c++ will be based on Ganymede which will include the quickly maturing Debug Services Framework. The DSF team has been busy joining the framework with CDT 5.0 and beefing up the reference implementation that works with gdb on linux.

The Carbide.c++ debugger uses the CDI apis to let our debug engine provide services to the common C++ debug support in CDT. That has worked pretty well but there are lots of new things we would like to do that would be a lot easier using DSF so I'm going to start looking into bringing up our debugger in this new, more flexible, and highly asynchronous environment. I haven't looked at DSF in any depth since last September and there has been a lot of refactoring since then but now that things have settled down with M5 I'm going to dive back in.

Bug Hunting in Big D

2008-02-11T14:00:00.000-06:00

The people in our beta group have been really positive while sending plenty of constructive criticism about things we need to improve. It's been very satisfying working with them to track down some of the bugs that escaped us last time. One was a report about debugger performance: some people said that when debugging they found that "stepping through code was really slow."

These kinds of problems can be really difficult to reproduce and fix because there are so many variables. Only a few people were reporting the problem and it wasn't consistent. We couldn't reproduce it at all.

A lot of things can impact debugger stepping performance including the speed of the debug connection and the number of variables displayed each time, but none seemed to be a factor in this case.

As people ran into the problem we started to get more specific information: we heard that not only was stepping in the debugger slow but the entire Eclipse environment became really slow. Then someone noticed that everything on their computer was slow: every application, menu etc.

Next we heard from a developer on one of Nokia's software teams in Dallas. He could reproduce the problem but it only happened when debugging a particular project. I setup the same project and tried the same steps but couldn't reproduce the problem.

A trip to Dallas from our office here in Austin can easily occupy a long day, much of it spent in traffic, so first we tried a screen sharing session. That confirmed I was following the same repro steps with the project but getting different results: my debug session hummed along while my Dallas colleague's ground to a halt with the Java VM process sucking up 80 - 90% of the cpu. But the most interesting thing was that the application he was debugging (a music player) was sending a huge amount of text to the console view. The entire time a song was playing the console view was flooded with debug output. After playing a few songs the audio began to skip and the big slow down began.

I pulled a few long songs from my music collection and while my console view was also flooded with various debug messages I didn't see any performance hit. Time for a road trip.

When I got to Dallas we quickly reproduced the problem and all the evidence pointed to the console output. The platform debug console has a setting that limits the amount of text in a console. My first theory was that we were overloading whatever method keeps the console under this limit. Maybe we were dumping in so much text it was thrashing about trying to keep the limit enforced?

Eclipse does so much cool stuff for you automatically I assumed the console limit worked that way too. But that's built into the debug platform's ProcessConsole and for various reasons ours is built on top of MessageConsole instead. So we didn't have any limit on the amount of text in a console. Over time all the text dumped into our console consumed more and more memory.

Limiting the amount of text in the console view was simple enough and has fixed the problem. I'm still not sure why the effect of the big memory leak on my Dallas colleague's computer was so severe while my similarly configured laptop wasn't bothered. In any event the whole experience shows how initial descriptions of bugs can be misleading and that you need work closely with the people using the tools all day in order to track down difficult to reproduce problems.

Carbide.c++ beta winding down

2008-01-24T15:50:00.000-06:00

Our Carbide.c++ tools for Symbian OS development have spent the last several months in front of hundreds of people in our beta program. Now it's time to wrap things up and get ready to release. For this update we moved to version 3.3.1 of the platform and 4.0.3 of CDT, added lots of little enhancements, fixed hundreds of bugs, and posted about a dozen patches for CDT and the platform.

People really like the improvements to the indexer and code navigation tools in CDT and gave us lots of feedback on where they could be improved, especially support for C++ templates. All the positive response was tempered by continued frustration with the way projects have to be organized so we are still looking for solutions to that problem.

Symbian OS projects have their own build system and our support for it is now better integrated into the new CDT project model. Our debugger now takes better advantage of the contextual launch support in Eclipse 3.3 and we added a view for managing and debugging lots of executables.

Opening up our bugzilla to everyone in the beta program has really been helpful and brought our team a little closer to everyone using the tools. Making our nightly builds available has let us get fixes out to people fast for quick validation. Overall I think running our beta program much like an Eclipse project has let us make the right kinds of changes faster than we have before.

Open Carbide.c++ beta begins

2007-11-26T16:10:00.000-06:00

Back at the CDT summit in late September someone asked me if I had gone into hiding. Not exactly, but along with the rest of the Carbide.c++ team here at Nokia I've been focused on getting a new update (v1.3) out to our beta community. We finished the first beta drop just before the Thanksgiving break so it's now out there for anyone to try out.

For this update to our C++ tools for Symbian OS we moved to a new version of the platform (3.3.x) and CDT (4.0.x). We are also reinventing our development process to be more agile and responsive with a lot of the inspiration for this coming from the Eclipse community: We're now running our beta program through a google group that is open to anyone who asks to join. Our internal bugzilla has been opened up so that everyone can use it to report and track bugs. We'll be doing periodic beta releases that have some testing behind them and also publishing the last week's worth of auto-tested nightly builds. The nightly builds will give us a way to validate bug fixes and get immediate feedback from the thrill seekers who don't want to wait for the next beta.

So far we're off to a good start with a couple hundred downloads and a couple dozen new bugs, mostly from people in the rest of the world that wasn't focused on turkey and football last week.

"I've got Barbies in my suitcase"

2007-09-29T19:10:00.000-06:00

I took a break from Carbide 1.3 development to attend last week's CDT summit in Ottawa. I wasn't sure what to expect going in but after people presented their prospective contributions we ended up with an exciting list of new work planned for CDT 5.0 next summer. There are a series of improvements to existing features like the parser, code navigation tools, editor, project model etc., and some new things like a class browser and the integration of the device debug project's DSF framework.

We also talked about EFS support in CDT and Doug is going to explore using this to allow projects to include an arbitrary set of files from anywhere in the file system. Not sure if it will work yet but this is a big issue for Carbide users and anyone else doing C/C++ so I'm encouraged by any attempt to fix it.

I got to share a lot of the feedback we have received from Carbide users and demo our new Executables View. The debug support in CDT is in transition and we had a good series of discussions about how to proceed. I've been exploring moving Carbide's debugger to use DSF and it was great to be able to talk to Pawel Piech (DSF lead) and the team from Ericsson doing the gdb/mi reference implementation. Pawel endured some good natured harassment about the current look of the DSF launch icon which finally prompted his shocking but unverified confession about the contents of his luggage.

On the flight back to Austin I noticed a guy a couple rows up running Eclipse on his laptop (It stood out in the sea of solitaire games, powerpoint slides and action movies on the other screens). Turned out to be Chris Aniszczyk so I finally got to meet and chat with him during the layover in Chicago.

Again I was impressed by the professional, pragmatic, and friendly bunch of people in the CDT community. Thanks to Doug, the QNX team, and the Eclipse folks for hosting the summit.

Usability and CDT 4.0

2007-06-12T07:55:00.000-06:00

When we first began developing our C++ tools we wanted to see what usability issues developers would face so we ran a series of testing sessions with people new to Eclipse, people who had used Eclipse for Java development, and people developing with Visual Studio.

The bug 154280 has more details but my story here is that the issues on the front line have been addressed in Europa.

When we asked people to build a newly created project they looked for a build button in the toolbar. (The platform has a Build All button but it's not easily identifiable and it builds all).

Now CDT 4.0 includes a Build Project button that builds the active configuration of the selected project and has a menu that lets you build any of the other ones.

When we asked people to debug their project they naturally hit the Debug button and ended up getting confused by the Launch Configuration dialog.

Now CDT 4.0 takes advantage of the new debug platform support for contextual launching: Hitting the debug button now does what people expect, launching a debug session for the selected project and using the launch shortcut to ask for any settings it needs along the way.

We're tracking lots of other usability issues from C/C++ developers but I'm pleased these two big out of the box problems are fixed in Europa.

Would you like JDT with that?

2007-05-14T15:55:00.000-06:00

In a recent conversation in bugzilla someone asked if it was common for people using CDT to also have the JDT installed. The short answer is "yes" but that's not always because they are developing in Java as well as C++.

When people start with our Carbide.c++ they run a branded installer like you get with most Windows apps and you get an entry in the Start menu etc. We install a JRE and a bunch of Eclipse features but not the JDT. People focused on C++ development didn't want to see "all the Java stuff" so we left it out.

But people who are also doing Java development (sometimes developing plug-ins for Carbide) or who already have Eclipse configured with their favorite plug-ins can just take our Carbide features and merge them in. Corporate IT people often like to have a standard version of Eclipse that usually includes the JDT since it is seen as part of the base platform. Also some plug-ins out there assume the JDT is present and will not work correctly if it isn't. So we've got to assume there is a good chance the JDT will show up at some point.

I recall one of the sessions at EclipseCon this year where the presenter was showing Eclipse with all of the Europa plug-ins (and maybe some others) installed. The UI was an impressive jumble of widgets and completely confusing. Someone said something about capabilities being useful but added that "no one uses them."

I was surprised as we use capabilities extensively to keep the user experience focused on C++ development when people are using Carbide, mostly by hiding CDT UI that isn't used when developing apps for Symbian OS. At some point I'll have to see if we could start including the JDT but keep the UI subdued unless you start working with Java projects.

Structuring C++ Projects in Eclipse

2007-05-07T14:36:00.000-06:00

We’ve been struggling with how to represent C++ projects in Eclipse so I thought I would share our story in case anyone can offer any better suggestions:

It’s common for developers to already have their code organized something like this:

/Source/Projects/CPPProjectOne/src – project one specific code
/Source/Projects/CPPProjectTwo/src – project two specific code
/Source/Projects/CPPProjectThree/src – project three specific code
/Source/Projects/Common/src – source common to all CPP projects

We can not copy the source files to another location or move anything around, we have to work with them exactly where they are in the file system. Since for each project we call a custom make builder that doesn’t depend on anything in Eclipse the main thing we want our Eclipse projects to do is contain as much of the code as possible so that all of the splendid IDE features work correctly.

Our first thought was to put the Eclipse projects in the workspace and then stuff them with linked resources to the actual source files and folders. That seemed like a great fit for linked resources and would be fairly straightforward to do. However we discovered that linked resources are not really first class citizens in the workspace: to begin with they don’t work with version control (41929) so we had to abandon that idea.

Our next try was to just locate the Eclipse project and it’s source root where the cpp project’s code actually lives. When we import a cpp project’s make file we can look at the included sources and determine a root location that covers all the files. So using the above example CPPProjectOne’s source root would be /Source/Projects. This works OK, we end up with an Eclipse project that lives at /Source/Projects/.project and includes resources for all of the desired files (plus lots of other stuff too).

The trouble begins when I want to work with CPPProjectTwo at the same time. I find that it has the same common source root location as CPPProjectOne, but I can’t put a second Eclipse project there as a .project file already exists in that location.

The best work around we have been able to think of for now is to have the second project use a more narrowly scoped source root, in this case it could use /Source/Projects/CPPProjectTwo. But that will leave other files it uses, like those from /Source/Projects/Common/src, out of the project.

If we could either put more than one Eclipse project in the same location or tell a project to use something other than its location as a source root (78438) then we could accommodate any number of projects. Or if linked resources worked better we could use those. Just now we’re stuck telling people to try the work around but we need to determine what direction to pursue longer term so we can make this work the way people expect.

Carbide 1.2 is Out the Door

2007-04-16T12:52:00.000-06:00

We've completed the final build for Carbide 1.2 so I can come up for air now. Carbide 1.2 is using Eclipse 3.2.1 and a modified version of CDT 4.0M5. Taking the risk on a pre-release version of CDT has worked out well: the code navigation features bring us much closer to what people expect and we've seen few problems. Over the last couple months our team has been helped by a great beta group of several hundred people testing Carbide in a production environment with lots of sizable real world projects. This has been illuminating and occasionally humbling as people told us what we had to deliver to pull them away from their current C++ development environment. We couldn't get all of it into this release and so we're left with a lot of raw feedback to distill into specific fixes and enhancements for next time.

Now I need to move ahead with work on my CDT bug list and planning for the next Carbide release which will use Eclipse 3.3 and the final version of CDT 4.0. How did it get to be mid April already?

Jolting News

2007-03-24T08:28:00.000-06:00

Since returning from EclipseCon I'm mostly had my head down focused on fixing bugs in the final few weeks before we ship Carbide.c++ 1.2. Then came the announcement last week that Carbide 1.1 had won the Jolt award for best mobile development tool! We had been runner up the year before but this time we beat out NetBeans and few others to capture the Jolt. Since we're all focused on the next release it's nice to get some recognition for the last one.

CDT 4.0 Demo at EclipseCon

2007-03-06T16:52:00.000-06:00

Doug Schaefer and Markus Schorn just finished a great demo of the new features in CDT 4.0 to a packed room. We're shipping our next version of Carbide with a customized version of CDT 4.0M5 and after a couple weeks of internal testing and a week in front of our beta testers things are looking very good. The improvements in code editing and navigation are really a big leap forward for CDT.

Automated UI Testing

2007-03-05T13:13:00.000-06:00

Our team has been adding automated UI testing to our JUnit tests over the past year. Since there seems to be growing interest in doing this I thought I would tell our story.

To begin with our developers wanted to run JUnit tests and our test team wanted to automate much of the testing they were doing by hand. It made sense to combine those efforts and have one set of tests everyone could run and contribute to. But we couldn't test everything we needed from JUnit without the ability to occasionally punch a button, enter text into a wizard, or choose a menu item.

We looked at tools that let you record user actions and then play them back. This makes a great demo and has immediate appeal to what many people initially think they want from automated UI testing. If you are a test team that is given a product and you just want to automate what would otherwise be a laborious manual test effort this is probably the best you can do. But tests constructed this way are often fragile and superficial: minor UI changes can break them and they don't really know what's going on underneath. We wanted our development and test efforts to be better integrated so we looked for a way to drive the UI from our JUnit tests.

We ended up using Abbot to do this with pretty good results. Abbot also has a script editor/recorder called Costello that doesn't work with SWT yet. I think this has led some people to think Abbot doesn't work with SWT at all, but we were interested in the stuff that we could call directly from our JUnit code and got that working right away.

Much of Abbot is pretty low-level and we ended up building a utility layer on top of it to handle common tasks and boilerplate code. This was easy to do and makes the code that uses it a lot cleaner.

A problem all UI testing tools have is identifying the exact widget to act on. If it doesn't have a unique name you have to do something fragile like reference it by relative position to known widgets. Even if it is something like a button that has a name like "Cancel" the test will break as soon as 'Cancel" is translated to another language so you'll need to translate that portion of the tests as well. The solution is to give significant SWT widgets unique IDs when you create them. This is easy to do so we started adding the ID to any UI that we create for Carbide. Of course that leaves the UI from the platform and CDT so there are still plenty of cases where we have to hunt around for the correct widget.

Our tests can drive the UI faster than any human could. This can lead to false problems with the test: it may click on a button in one view and then want to look at an expanded tree item in another view but the tree doesn't expand instantly. Events often have to ripple through the platform before the UI settles down. There are places where you may need to insert a pause to make the test work reliably. This has to be done judiciously because there may be someone out there who downs enough coffee to turn what seems like unlikely behavior into a real bug.

With the addition of Abbot we've been able to write tests that automate much of the debugger in CDT. While some of the tests in CDT are updated and run with every build the debugger tests are out of date and wouldn't provide the same level of coverage. I'd like to get our general debugger tests into CDT as most things can be tested the same way regardless of the debugger engine on the back-end.

I've glossed over a lot of detail but I think that when looking at automated UI testing tools it's best to begin by deciding if you want to integrate them into a test driven development cycle. In many organizations testing is a completely segregated function done by people who aren't in a position to collaborate with developers on JUnit tests and so one of the record/playback/scripting tools is probably the best option. But for Carbide.c++ we've been able to use Abbot to extend the reach of our JUnit tests for a more integrated and comprehensive approach to testing and development. Now we just need to build more tests...

Introduction

2007-03-01T23:11:00.000-06:00

Welcome to my Nokia Carbide on Eclipse Blog. So who am I and what's going on with Nokia, Carbide, and Eclipse?

I'm a software developer on the C++ tools team at Nokia. Nokia makes a ton of the little ringing vibrating things people call cell/mobile phones or more grandly “mobile devices”. A large and growing number of these are smartphones, little computers with an operating system, storage, user interface, and lots of connectivity. Most smartphones run Symbian OS with additional platform and UI services (S60 or UIQ) on top. It’s a real software platform that lets you install and run additional applications.

These apps are built by software developers who need good tools. Carbide.c++ is Nokia’s C++ tool set for mobile software development. Carbide is used to produce mobile applications by the companies that make the phones and by the developer community that supports them.

Carbide is built on Eclipse: our team takes the latest platform release, EMF, GEF, usually a pre-release version of CDT, plus a bit of DSDP-DD, then adds in a build system, debugger, user interface designer, performance tools, and documentation to produce a commercial product. As we develop Carbide our team contributes back to the Eclipse community. Nokia is a Eclipse Foundation strategic developer. I’m a committer on the CDT project.

I’m looking forward to sharing our team’s experiences building Carbide and developing with Eclipse.