Carbide 1.3 End Game

We are in the last weeks of testing Carbide 1.3: the code is frozen, the test team is rechecking everything, and the beta testers are trying out the release candidate build. This is usually the time when some bug that has been around all along makes a surprise appearance in a really troubling way.

Last week we started getting some reports of the entire workbench locking up. People who were really beating on the tools all day said that often after a paste or save operation the UI would be unresponsive. Of course we couldn't reproduce the problem but we had two interesting clues: some people said the UI started responding again after a while and one beta tester sent in a screenshot of the frozen workbench.

Everyone has their own personal "time out" setting: the amount of time they are willing to wait for an unresponsive application to start working again before they kill it or restart their computer or chuck it over the cube wall. So that some people said the workbench came back to life after a while probably indicated we had some lengthly process blocking the UI thread, not that Carbide was frozen forever.

The screenshot gave us a few more clues: the progress indicator said the CDT indexer job was just starting and that the text the user was pasting hadn't yet appeared in the editor. I started trying to reproduce the problem by forcing a project to reindex and then repeatedly pasting code.

After a lot of pasting we found that the problem had nothing to do with the paste operation but with the use of the control key (as in ctrl-v to paste or ctrl-s to save). With the control key down CDT was asked to produce a hyperlink. Normally this happens quickly because the file has already been indexed and the results cached, but when it's dirty it proceeds to do some indexing on the UI thread which leaves the workbench completely unresponsive.

It seemed like the right thing to do in this case was to just not do the indexing if the cache was stale. We turned to the CDT team to confirm this and within a few hours we had changes in place to prevent this problem and a couple other related issues too.

So now I'm starting this week like I did the last: hoping Carbide 1.3 can slide out the door without any serious issues popping up.

Looking at DSF

Our next major release of Carbide.c++ will be based on Ganymede which will include the quickly maturing Debug Services Framework. The DSF team has been busy joining the framework with CDT 5.0 and beefing up the reference implementation that works with gdb on linux.

The Carbide.c++ debugger uses the CDI apis to let our debug engine provide services to the common C++ debug support in CDT. That has worked pretty well but there are lots of new things we would like to do that would be a lot easier using DSF so I'm going to start looking into bringing up our debugger in this new, more flexible, and highly asynchronous environment. I haven't looked at DSF in any depth since last September and there has been a lot of refactoring since then but now that things have settled down with M5 I'm going to dive back in.

Bug Hunting in Big D

The people in our beta group have been really positive while sending plenty of constructive criticism about things we need to improve. It's been very satisfying working with them to track down some of the bugs that escaped us last time. One was a report about debugger performance: some people said that when debugging they found that "stepping through code was really slow."

These kinds of problems can be really difficult to reproduce and fix because there are so many variables. Only a few people were reporting the problem and it wasn't consistent. We couldn't reproduce it at all.

A lot of things can impact debugger stepping performance including the speed of the debug connection and the number of variables displayed each time, but none seemed to be a factor in this case.

As people ran into the problem we started to get more specific information: we heard that not only was stepping in the debugger slow but the entire Eclipse environment became really slow. Then someone noticed that everything on their computer was slow: every application, menu etc.

Next we heard from a developer on one of Nokia's software teams in Dallas. He could reproduce the problem but it only happened when debugging a particular project. I setup the same project and tried the same steps but couldn't reproduce the problem.

A trip to Dallas from our office here in Austin can easily occupy a long day, much of it spent in traffic, so first we tried a screen sharing session. That confirmed I was following the same repro steps with the project but getting different results: my debug session hummed along while my Dallas colleague's ground to a halt with the Java VM process sucking up 80 - 90% of the cpu. But the most interesting thing was that the application he was debugging (a music player) was sending a huge amount of text to the console view. The entire time a song was playing the console view was flooded with debug output. After playing a few songs the audio began to skip and the big slow down began.

I pulled a few long songs from my music collection and while my console view was also flooded with various debug messages I didn't see any performance hit. Time for a road trip.

When I got to Dallas we quickly reproduced the problem and all the evidence pointed to the console output. The platform debug console has a setting that limits the amount of text in a console. My first theory was that we were overloading whatever method keeps the console under this limit. Maybe we were dumping in so much text it was thrashing about trying to keep the limit enforced?

Eclipse does so much cool stuff for you automatically I assumed the console limit worked that way too. But that's built into the debug platform's ProcessConsole and for various reasons ours is built on top of MessageConsole instead. So we didn't have any limit on the amount of text in a console. Over time all the text dumped into our console consumed more and more memory.

Limiting the amount of text in the console view was simple enough and has fixed the problem. I'm still not sure why the effect of the big memory leak on my Dallas colleague's computer was so severe while my similarly configured laptop wasn't bothered. In any event the whole experience shows how initial descriptions of bugs can be misleading and that you need work closely with the people using the tools all day in order to track down difficult to reproduce problems.

Carbide.c++ beta winding down

Our Carbide.c++ tools for Symbian OS development have spent the last several months in front of hundreds of people in our beta program. Now it's time to wrap things up and get ready to release. For this update we moved to version 3.3.1 of the platform and 4.0.3 of CDT, added lots of little enhancements, fixed hundreds of bugs, and posted about a dozen patches for CDT and the platform.

People really like the improvements to the indexer and code navigation tools in CDT and gave us lots of feedback on where they could be improved, especially support for C++ templates. All the positive response was tempered by continued frustration with the way projects have to be organized so we are still looking for solutions to that problem.

Symbian OS projects have their own build system and our support for it is now better integrated into the new CDT project model. Our debugger now takes better advantage of the contextual launch support in Eclipse 3.3 and we added a view for managing and debugging lots of executables.

Opening up our bugzilla to everyone in the beta program has really been helpful and brought our team a little closer to everyone using the tools. Making our nightly builds available has let us get fixes out to people fast for quick validation. Overall I think running our beta program much like an Eclipse project has let us make the right kinds of changes faster than we have before.

Open Carbide.c++ beta begins

Back at the CDT summit in late September someone asked me if I had gone into hiding. Not exactly, but along with the rest of the Carbide.c++ team here at Nokia I've been focused on getting a new update (v1.3) out to our beta community. We finished the first beta drop just before the Thanksgiving break so it's now out there for anyone to try out.

For this update to our C++ tools for Symbian OS we moved to a new version of the platform (3.3.x) and CDT (4.0.x). We are also reinventing our development process to be more agile and responsive with a lot of the inspiration for this coming from the Eclipse community: We're now running our beta program through a google group that is open to anyone who asks to join. Our internal bugzilla has been opened up so that everyone can use it to report and track bugs. We'll be doing periodic beta releases that have some testing behind them and also publishing the last week's worth of auto-tested nightly builds. The nightly builds will give us a way to validate bug fixes and get immediate feedback from the thrill seekers who don't want to wait for the next beta.

So far we're off to a good start with a couple hundred downloads and a couple dozen new bugs, mostly from people in the rest of the world that wasn't focused on turkey and football last week.

"I've got Barbies in my suitcase"

I took a break from Carbide 1.3 development to attend last week's CDT summit in Ottawa. I wasn't sure what to expect going in but after people presented their prospective contributions we ended up with an exciting list of new work planned for CDT 5.0 next summer. There are a series of improvements to existing features like the parser, code navigation tools, editor, project model etc., and some new things like a class browser and the integration of the device debug project's DSF framework.

We also talked about EFS support in CDT and Doug is going to explore using this to allow projects to include an arbitrary set of files from anywhere in the file system. Not sure if it will work yet but this is a big issue for Carbide users and anyone else doing C/C++ so I'm encouraged by any attempt to fix it.

I got to share a lot of the feedback we have received from Carbide users and demo our new Executables View. The debug support in CDT is in transition and we had a good series of discussions about how to proceed. I've been exploring moving Carbide's debugger to use DSF and it was great to be able to talk to Pawel Piech (DSF lead) and the team from Ericsson doing the gdb/mi reference implementation. Pawel endured some good natured harassment about the current look of the DSF launch icon which finally prompted his shocking but unverified confession about the contents of his luggage.

On the flight back to Austin I noticed a guy a couple rows up running Eclipse on his laptop (It stood out in the sea of solitaire games, powerpoint slides and action movies on the other screens). Turned out to be Chris Aniszczyk so I finally got to meet and chat with him during the layover in Chicago.

Again I was impressed by the professional, pragmatic, and friendly bunch of people in the CDT community. Thanks to Doug, the QNX team, and the Eclipse folks for hosting the summit.

Usability and CDT 4.0

When we first began developing our C++ tools we wanted to see what usability issues developers would face so we ran a series of testing sessions with people new to Eclipse, people who had used Eclipse for Java development, and people developing with Visual Studio.

The bug 154280 has more details but my story here is that the issues on the front line have been addressed in Europa.

When we asked people to build a newly created project they looked for a build button in the toolbar. (The platform has a Build All button but it's not easily identifiable and it builds all).

Now CDT 4.0 includes a Build Project button that builds the active configuration of the selected project and has a menu that lets you build any of the other ones.


When we asked people to debug their project they naturally hit the Debug button and ended up getting confused by the Launch Configuration dialog.

Now CDT 4.0 takes advantage of the new debug platform support for contextual launching: Hitting the debug button now does what people expect, launching a debug session for the selected project and using the launch shortcut to ask for any settings it needs along the way.

We're tracking lots of other usability issues from C/C++ developers but I'm pleased these two big out of the box problems are fixed in Europa.