Things to look for in an automation tool:
- Object recognition – How does the tool recognize elements in your application under test? Does it simply rely on image based testing or can it individually identify each element and access/set values? If so, does it recognize all of your controls, even custom ones?
- Object identification – How does the tool identify elements? Can you uniquely identify several similar elements? How fine grained is the search method? Can you search using xpath, regex etc?
2. Performance and Reliability:
- Performance – How long does the tool take to identify elements?
- Reliability – How easy is it to create reliable scripts which run over and over without a failure of the script?
- Stability – How long has the tool been around for? Is the tool still being developed and being kept up to date with the latest technologies, for example Firefoxs crazy releases?
3. Test Development:
- Platforms – Which Operating Systems/Device types are supported by the tool.
- Language – What languages can tests be written in? Does the language support threading, file manipulation, database support, xml manipulation, service interaction, registry interaction etc?
- Test control – Can you control test execution from the tool? Can you create test suites? Can you run from command line? Can you remotely execute tests? Can you use other test frameworks such as Nunit?
- Screen shots -Can the tool report screen shots of your application under test? …of the whole desktop?…of individual elements?
- Test Data – Does the tool support functionality to bind test data to test scripts?
4. Test Code integration:
- Version control – Does the tool provide script versioning or can you use industry versioning systems such as SubVersion.
- Continuous Integration – Can you set the tool up with a CI system such as Cruise Control, Hudson or Jenkins?
- Re-Use – Can you reuse parts of scripts in other test projects for other products you are testing?
- Runtime – What are the requirements to run tests on different machines?
- Reporting – Does the tool generate reports or do you have to add your own custom reporting library?
5. Cost and Support:
- Cost – Is the tool free or does it cost? If it costs, what is the licensing model?
- Learning curve – What help/tutorials does the tool provide? How easy is the tool to use? Does it use a mainstream language such as .NET or Java, in which case there is plenty of official documentation, forums, help etc online to help you. Developers in your office may also be able to help you
- Support – How comprehensive is the support for the tool? How quick do support get back to you? How quick will bugs in the tool be fixed (if any)? Is there a voting system to decide which bugs get fixed?
- Web support – Can the tool test websites? Which browsers does it support?
- Native Support – Can the tool test applications running natively on your host? Make sure you plan for the future.
- Known Issues – Look around forums, are there any common known issues?