Saturday, March 24, 2007
Jolting News
Since returning from EclipseCon I'm mostly had my head down focused on fixing bugs in the final few weeks before we ship Carbide.c++ 1.2. Then came the announcement last week that Carbide 1.1 had won the Jolt award for best mobile development tool! We had been runner up the year before but this time we beat out NetBeans and few others to capture the Jolt. Since we're all focused on the next release it's nice to get some recognition for the last one.
Tuesday, March 6, 2007
CDT 4.0 Demo at EclipseCon
Doug Schaefer and Markus Schorn just finished a great demo of the new features in CDT 4.0 to a packed room. We're shipping our next version of Carbide with a customized version of CDT 4.0M5 and after a couple weeks of internal testing and a week in front of our beta testers things are looking very good. The improvements in code editing and navigation are really a big leap forward for CDT.
Monday, March 5, 2007
Automated UI Testing
Our team has been adding automated UI testing to our JUnit tests over the past year. Since there seems to be growing interest in doing this I thought I would tell our story.
To begin with our developers wanted to run JUnit tests and our test team wanted to automate much of the testing they were doing by hand. It made sense to combine those efforts and have one set of tests everyone could run and contribute to. But we couldn't test everything we needed from JUnit without the ability to occasionally punch a button, enter text into a wizard, or choose a menu item.
We looked at tools that let you record user actions and then play them back. This makes a great demo and has immediate appeal to what many people initially think they want from automated UI testing. If you are a test team that is given a product and you just want to automate what would otherwise be a laborious manual test effort this is probably the best you can do. But tests constructed this way are often fragile and superficial: minor UI changes can break them and they don't really know what's going on underneath. We wanted our development and test efforts to be better integrated so we looked for a way to drive the UI from our JUnit tests.
We ended up using Abbot to do this with pretty good results. Abbot also has a script editor/recorder called Costello that doesn't work with SWT yet. I think this has led some people to think Abbot doesn't work with SWT at all, but we were interested in the stuff that we could call directly from our JUnit code and got that working right away.
Much of Abbot is pretty low-level and we ended up building a utility layer on top of it to handle common tasks and boilerplate code. This was easy to do and makes the code that uses it a lot cleaner.
A problem all UI testing tools have is identifying the exact widget to act on. If it doesn't have a unique name you have to do something fragile like reference it by relative position to known widgets. Even if it is something like a button that has a name like "Cancel" the test will break as soon as 'Cancel" is translated to another language so you'll need to translate that portion of the tests as well. The solution is to give significant SWT widgets unique IDs when you create them. This is easy to do so we started adding the ID to any UI that we create for Carbide. Of course that leaves the UI from the platform and CDT so there are still plenty of cases where we have to hunt around for the correct widget.
Our tests can drive the UI faster than any human could. This can lead to false problems with the test: it may click on a button in one view and then want to look at an expanded tree item in another view but the tree doesn't expand instantly. Events often have to ripple through the platform before the UI settles down. There are places where you may need to insert a pause to make the test work reliably. This has to be done judiciously because there may be someone out there who downs enough coffee to turn what seems like unlikely behavior into a real bug.
With the addition of Abbot we've been able to write tests that automate much of the debugger in CDT. While some of the tests in CDT are updated and run with every build the debugger tests are out of date and wouldn't provide the same level of coverage. I'd like to get our general debugger tests into CDT as most things can be tested the same way regardless of the debugger engine on the back-end.
I've glossed over a lot of detail but I think that when looking at automated UI testing tools it's best to begin by deciding if you want to integrate them into a test driven development cycle. In many organizations testing is a completely segregated function done by people who aren't in a position to collaborate with developers on JUnit tests and so one of the record/playback/scripting tools is probably the best option. But for Carbide.c++ we've been able to use Abbot to extend the reach of our JUnit tests for a more integrated and comprehensive approach to testing and development. Now we just need to build more tests...
To begin with our developers wanted to run JUnit tests and our test team wanted to automate much of the testing they were doing by hand. It made sense to combine those efforts and have one set of tests everyone could run and contribute to. But we couldn't test everything we needed from JUnit without the ability to occasionally punch a button, enter text into a wizard, or choose a menu item.
We looked at tools that let you record user actions and then play them back. This makes a great demo and has immediate appeal to what many people initially think they want from automated UI testing. If you are a test team that is given a product and you just want to automate what would otherwise be a laborious manual test effort this is probably the best you can do. But tests constructed this way are often fragile and superficial: minor UI changes can break them and they don't really know what's going on underneath. We wanted our development and test efforts to be better integrated so we looked for a way to drive the UI from our JUnit tests.
We ended up using Abbot to do this with pretty good results. Abbot also has a script editor/recorder called Costello that doesn't work with SWT yet. I think this has led some people to think Abbot doesn't work with SWT at all, but we were interested in the stuff that we could call directly from our JUnit code and got that working right away.
Much of Abbot is pretty low-level and we ended up building a utility layer on top of it to handle common tasks and boilerplate code. This was easy to do and makes the code that uses it a lot cleaner.
A problem all UI testing tools have is identifying the exact widget to act on. If it doesn't have a unique name you have to do something fragile like reference it by relative position to known widgets. Even if it is something like a button that has a name like "Cancel" the test will break as soon as 'Cancel" is translated to another language so you'll need to translate that portion of the tests as well. The solution is to give significant SWT widgets unique IDs when you create them. This is easy to do so we started adding the ID to any UI that we create for Carbide. Of course that leaves the UI from the platform and CDT so there are still plenty of cases where we have to hunt around for the correct widget.
Our tests can drive the UI faster than any human could. This can lead to false problems with the test: it may click on a button in one view and then want to look at an expanded tree item in another view but the tree doesn't expand instantly. Events often have to ripple through the platform before the UI settles down. There are places where you may need to insert a pause to make the test work reliably. This has to be done judiciously because there may be someone out there who downs enough coffee to turn what seems like unlikely behavior into a real bug.
With the addition of Abbot we've been able to write tests that automate much of the debugger in CDT. While some of the tests in CDT are updated and run with every build the debugger tests are out of date and wouldn't provide the same level of coverage. I'd like to get our general debugger tests into CDT as most things can be tested the same way regardless of the debugger engine on the back-end.
I've glossed over a lot of detail but I think that when looking at automated UI testing tools it's best to begin by deciding if you want to integrate them into a test driven development cycle. In many organizations testing is a completely segregated function done by people who aren't in a position to collaborate with developers on JUnit tests and so one of the record/playback/scripting tools is probably the best option. But for Carbide.c++ we've been able to use Abbot to extend the reach of our JUnit tests for a more integrated and comprehensive approach to testing and development. Now we just need to build more tests...
Thursday, March 1, 2007
Introduction
Welcome to my Nokia Carbide on Eclipse Blog. So who am I and what's going on with Nokia, Carbide, and Eclipse?
I'm a software developer on the C++ tools team at Nokia. Nokia makes a ton of the little ringing vibrating things people call cell/mobile phones or more grandly “mobile devices”. A large and growing number of these are smartphones, little computers with an operating system, storage, user interface, and lots of connectivity. Most smartphones run Symbian OS with additional platform and UI services (S60 or UIQ) on top. It’s a real software platform that lets you install and run additional applications.
These apps are built by software developers who need good tools. Carbide.c++ is Nokia’s C++ tool set for mobile software development. Carbide is used to produce mobile applications by the companies that make the phones and by the developer community that supports them.
Carbide is built on Eclipse: our team takes the latest platform release, EMF, GEF, usually a pre-release version of CDT, plus a bit of DSDP-DD, then adds in a build system, debugger, user interface designer, performance tools, and documentation to produce a commercial product. As we develop Carbide our team contributes back to the Eclipse community. Nokia is a Eclipse Foundation strategic developer. I’m a committer on the CDT project.
I’m looking forward to sharing our team’s experiences building Carbide and developing with Eclipse.
I'm a software developer on the C++ tools team at Nokia. Nokia makes a ton of the little ringing vibrating things people call cell/mobile phones or more grandly “mobile devices”. A large and growing number of these are smartphones, little computers with an operating system, storage, user interface, and lots of connectivity. Most smartphones run Symbian OS with additional platform and UI services (S60 or UIQ) on top. It’s a real software platform that lets you install and run additional applications.
These apps are built by software developers who need good tools. Carbide.c++ is Nokia’s C++ tool set for mobile software development. Carbide is used to produce mobile applications by the companies that make the phones and by the developer community that supports them.
Carbide is built on Eclipse: our team takes the latest platform release, EMF, GEF, usually a pre-release version of CDT, plus a bit of DSDP-DD, then adds in a build system, debugger, user interface designer, performance tools, and documentation to produce a commercial product. As we develop Carbide our team contributes back to the Eclipse community. Nokia is a Eclipse Foundation strategic developer. I’m a committer on the CDT project.
I’m looking forward to sharing our team’s experiences building Carbide and developing with Eclipse.
Subscribe to:
Posts (Atom)
