Components of a good bug report

A good bug report can improve over all communication of an Agile process. For e.g. A good bug report can help:

    • Managers in triaging bugs faster
    • Developers in reproducing the bugs faster
    • QA in verification of fixed bugs faster
    • Business Analyst in providing bug/feature clarification faster
    • Technical support in updating the customer about the open bugs faster

To submit a good report is quite tedious. It takes discipline experience and a solid checklist to write a good bug report every time a bug is found. Here is a quick checklist what a good bug report may contain: 

    • A summary/header
    • A clear description
    • Step by step recreation instructions
    • What error the steps produced
    • What were the actual results
    • Screen captures
    • Error and debugging logs
    • Setting a priority
    • Setting a critical status
    • Assigning the issue to the proper person or group
    • The area of the application/system where the issue was found.
    • The type of issue found
    • Linking to other relevant issues

And the list can go on depending on unique requirements of the company or application under test.

Author – Madhu Jain

Test Automation tools comparison – Selenium vs Sahi vs Watir

Automated software testing tool allows users to automate testing. Testing tool for web applications provides framework for software test developers to develop test cases for testing applications. Testing is achieved by simulating mouse clicks, keyboard entries and other user workflows.

 

There are lots of tools available commercially as well as open source. To keep my clients’ project cost low, I prefer using open source tools. Generally, open source tools also have large community support compared to the tools with a price tag. Some of the popular test automation tools are selenium, Sahi, tool with Watir webdriver (Cucumber). Below is a comparison between three of my all time favorite tools J

 

  Selenium Sahi Watir
Browser Works only on Firefox Multi browser
  • Multi browser
Frames and pop-up support Has trouble recording IFrames, Frames and popup windows Can record on IFrames, Frames and popup windows Frames and pop-ups are accessible using API
  For Frames and IFrames, need to explicitly selectFrame Implicit smart identification of elements even across Frames and IFrames Identification can be done via titles or text
  Uses XPath for identification of elements if id or name not present Uses various algorithms to uniquely identify elements in a simple human recognizable way Use simple algorithms – easy to debug
Programming Language support Java, Ruby, Perl, Python, C# (and may be more). Sahi Script, Java, Ruby Sahi Script has the syntax of javascript but can interoperate with any Java code directly from script. The Java/Ruby drivers are available since Sahi V3 Ruby
  Needs language bridges for each new feature. For example, needs java bridge to invoke Flash via ExternalAPI. Sahi Script can directly invoke anything exposed by javascript. Rich API to invoke any function
Ease of use Easy to get started with because of Selenium IDE which is a firefox plugin. Estimated start time less than 5 minutes Relatively steeper learning curve compared to Selenium – it needs installation of Sahi. Estimated start time 10-30 minutes, depending on Java installation etc. Relatively shorter learning cureve – requires Ruby and rubygems
  Deep learning curve to switch from Selenium IDE to Selenium RC. There is only one mode of operation for Sahi. Extremely simple to learn and use for testers Needs Ruby/System programming knowledge to allow distribution of tasks
  Knowledge of programming language required Can achieve most automation with just functions and variables. Sahi has inbuilt APIs for most other complex tasks like data driven testing Knowledge of Ruby programming language required
  Needs JUnit style of programming Can choose your own style Ruby style
  Uses XPath based identification for elements in complex html structures or those with dynamic ids. css selectors and javascript may also be used. Has nearness APIs like _in and _near which can help show nearness of elements. Eg. _image(“delete.gif”, _near(“username 4”)) Built in support for complex functions
  Needs waits for making AJAX work No waits needed in 90% cases No wait needed
  Supports parallel execution Inbuilt parallel execution. Needs only one parameter change Supports parallel execution outside Watir/Ruby

 

 

 

Cucumber tips and tricks

If you have just started cuking, you will find the below post helpful.

What is cucumber (in context to software testing)?

Cucumber is a ruby based tool for tomated testing framework development. It is different than other testing tools as it lets you execute plain-text functional descriptions (features) as automated tests. This tools bridges the technical gap between business analysts, developers and QA developers by allowing them to execute the user story requirements as test scenarios in plan English. This type of testing/development is also known as Behavior Driven Development (BDD).

Over the past few years, Cucumber has been one of my favorite tools for developing BDD based automated testing frameworks and test suites. I have used this tool for testing back-end as well front-end of applications.

Here is couple of tips and tricks that I use everyday while cuking:

  1. Use ‘–expand’ to expand Scenario outline tables in Cucumber output: The default cucumber option is to not expand the scenario outline table in the reporting. So when you run bundle exec features, it will print out the report like the one below:
 Scenario Outline: Multiply two numbers  
   Given I have entered "<num_1>" into the calculator
   And I press the multiply button  
   And I have entered "<num_2>" into the calculator
   And I press the equal button
   Then the result should be "<result>" on the screen
  Examples:
    |num_1 | num_2 | result |
    | 1    |   1   |   1    |  
    | 2    |   2   |   4    | 
    | 3    |   5   |   15   |

Use ‘–expand’ options to expand the scenario outline for better verbose reporting.

Here is the result of running bundle exec –expand cucumber features:

   Scenario Outline: Multiply two numbers                   
   Given I have entered "<num_1>" into the calculator
   And I press the multiply button                       
   And I have entered "<num_2>" into the calculator 
   And I press the equal button                     
   Then the result should be "" on the screen

   Examples:

     Scenario: | 1 | 1 | 1 |                   
       Given I have entered "1" into the calculator
       And I press the add button                 
       And I have entered "1" into the calculator 
       And I press the equal button               
       Then the result should be "1" on the screen

     Scenario: | 2 | 2 | 4 |                   
       Given I have entered "2" into the calculator
       And I press the add button                 
       And I have entered "2" into the calculator 
       And I press the equal button               
       Then the result should be "4" on the screen

     Scenario: | 3 | 5 | 15 |                   
       Given I have entered "3" into the calculator
       And I press the add button                 
       And I have entered "5" into the calculator 
       And I press the equal button               
       Then the result should be "15" on the screen

2. Regular expression cheat sheet for cucumber – Regular expressions are the key to Cucumber’s flexibility. Well-crafted regular expressions let you reuse step definitions, avoiding duplication and keeping your tests maintainable.

  1. Anchors: The regular expression I’m logged in matches I’m logged in and I’m logged in as an admin. To avoid ambiguous matches, use ^I’m logged in$. The caret at the beginning anchors to the beginning of the string. The dollar at the end does the same with the end of the string. Use these with all your step definitions and you won’t have surprise matches.
  2. Wildcards and quantifiers:
.* matches anything (or nothing), literally “any character (except a newline) 0 or more times”
.+ matches at least one of anything
[0-9]* or d* matches a series of digits (or nothing)
[0-9]+ or d+ matches one or more digits
“[^”]*” matches something (or nothing) in double quotes
an? matches a or an (the question mark makes the n optional)

3. Use Tags: TAGGING SCENARIOS

Another handy Cucumber feature is the ability to tag scenarios. Tagging a scenario is achieved by simply adding @tagname above the scenario definition, as such:

 Scenario: User is not signed up   
Given no user exists with an email of "email@person.com"  
When I go to the sign in page  
And I sign in as "email@person.com/password"  
Then I should see "Bad email or password" 
And I should not be signed in

To run scenarios with a specific tag:

cucumber --tags@tagname

To exclude scenarios with a certain tag, prefix the tag with a tilde:

cucumber --tags ~@wip

A common tag name that Cucumber supports out of the box is @wip (work in progress) and provides rake tasks to accommodate using this pattern:

rake cucumber:wip # runs only scenarios with a wip tag
rake cucumber:ok # runs all scenarios without a wip tag

Testing a Legacy product – Get busy in creating a TEST PLAN first!

Have you been ever hired to test the legacy product? Every organization, from start-ups to fortune 500s, has their own definition of legacy product. From a QA perspective I would define a legacy product as a product with NO formal written requirements. The product documentation is known as requirements’ document.

So, where do you start when it comes to testing a product with no requirements?

What Are We Testing?

I don’t believe there is any one right way to approach to test a product like that. However all the different methodologies ultimately take us in the direction of identifying a similar starting point – “A game plan”. A plan that indicates the success measures when we start testing the product. We need to document testing requirements and call it a Test Plan.

What is a good test plan and how to build it?

A good test plan should indicate testing goals along with any pass or fail conditions for every product requirement. A good test plan should include a lot of additional information – schedules, roles of team members, tools to be used, etc. But how do you start building a test plan when you have no defined goals and product requirements.

The test plan could be broken up into two pieces:

1)    What is the activity that the end users of the application do? (What are our use-cases?)

2)    What are the testing benchmark requirements? 

Identify Use Cases

With the brand new application in your hands, how can you determine what a user is supposed to do with it? There are several approaches:

1)    The best approach is usually to ask someone in product marketing or product management – someone who has done the work to create the user scenarios that the development team has implemented.

2)    Ask for a Marketing (or Product) Requirements Document that has this information documented.

3)    Check the product log files captured in the application’s infrastructure. Ask your web or application server administrative team if this is information they capture. For e.g. they may use tools like Omniture, Coremetrics, Web Trends or Google Analytics that can help with performance use case.

4)    If there are no logs, it’s usually possible to turn on some sort of logging for a period of time (a day or two perhaps) so that tracing user activity can be captured.

5)    You could look into getting access to actual end-users by talking to internal resource – customer or technical support teams, sales representative, or (again) marketing folks.

6)    If everything else fails use your common sense. Take a look at the application itself and decide what it is that YOU might do if you were an end-user. If you’re testing an online store type of application, it’s a pretty good bet that users are going to browse through the product catalog, add items to a shopping cart and make a purchase or two. Online banking customers are probably checking their balance and paying bills…you get the idea.

Generally speaking, you should not waste time trying to identify every single use case through the application since the bulk of user traffic will be captured in only a few transactions. Keep in mind the famous “80/20 rule”; that is 20% of the transactions cover 80% of application’s core functionality.

Establish Pass/Fail Requirements

Now we have established the use cases/product requirements – it is important to list the detailed pass or fail criteria. For. e.g. you might want to establish benchmark requirements for performance testing such as are you measuring response time or load time? How fast should be the page load up time for any given number of concurrent users? Now how do you validate your benchmark requirements?

1)    Involve various stakeholders such as product management, marketing, business analysts, etc.

2)    Crosscheck with contractual Service Level Agreements in place between your company and a customer, or between teams within your organization.

Now it’s time to execute your game plan and start reporting those bugs. Remember a good test plan goes a long way! Who knows one day your test plan becomes “the only trusted document” for their legacy product!

Author: Madhu Jain (An explorer)

Kaizen Testing

A Facebook UI bug

Spotted another Facebook UI bug since one of my previous posts. UI bugs are not generally considered show stoppers. However if Usability and User Interface are your strongest product assets, these bugs should be considered release blockers. I am wondering if Facebook`s QA team ever reproduced this issue in-house!

Image

Agile Test Automation is Incomplete Without Continuous Integration

Test Automation and Continuous Integration (CI) go hand in hand. In Agile world quick feedback is critical. Test Automation provides feedback about the quality of developed code and CI can accelerate that feedback!

What is Continuous Integration?

Continuous Integration is a software development practice where members of a team integrate their work frequently; usually each person integrates at least daily, leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. Martin Fowler

Continuous integration is an ongoing and rapid process. This means that teams need to be able to react quickly when a build breaks or new tests need to be built. In order to integrate a quality code into existing features, it is important to validate it against existing tests. If the tests fail, the code (build) should be rejected until either the bugs are fixed or the test cases are updated.

Below are the steps to deploy a quality code using automated tests and CI:

1. Use source code repository (Version Control System) – It is the first and required step to start using continuous integration. For e.g. Github, CVS, Perforce, SVN

  • Introduce check-in policy – Check in policy may be introduced after establishing source code repository. This depends on the team and length of the sprints. Firstly it involves storing all necessary files in repository. Secondly, each developer should check-in as often as possible completed part of work. Frequently integration show problem early and either solution is easy or rollback is necessary for only small part of solution.

2. Automate build – Next step is to perform build automatically – no more than one action should be required to start it. Either developer should start it manually or the build is automatically triggered after each code check-in (commit). The build process could be scripted using Ant, CruiseControl etc.

3. Create auto-deployable test environment and use automated test suite – After each build new version of binaries should be automatically deployed to test server. After each deployment, kick off automated tests to validate the committed code against existing features. The automated tests act as gate keeper for new code before it is integrated. It is time consuming to run source code validation but saves time during manual/regression testing and improves over all code quality. A solid automation test suite acts as a safety net and filters out the buggy code to some level.

4. Report and integrate – Report the status of automated testing. Having a pass/fail label can help in rejecting the buggy code.

  • Introduce re-submission policy: Developers need to investigate the failures with the highest priority. They either should fix the bugs or update the test cases and go back to step 1 for re-submission. Once all the tests pass, the quality code should be integrated to the master repository and deployed to staging for further testing.

Take charge and be curious!

Do you wait for the feature to be delivered before testing or test a product when it is being developed?

If a Quality Assurance Analyst wants to contribute to quality, they can do more than testing a product when it is delivered. QAs have many other duties during software development. They know the product very well from regression testing. They are aware of the various test input data sets. They understand the changing business needs and the requirements. They facilitate in environment and data preparation. They challenge the architecture and analysis, coach developers and define acceptance criteria to check their delivered product effectively on standards.  A tester understands and contributes to the release and deployment mechanisms. Over all more than anyone ‘testers are aware of products’ risks’.

So, DO NOT under estimate your position in flagging risky feature developments! Contribute in delivering high quality products by participating right from the beginning.

Product Development Phase

How can a QA rise and shine?
Sprint Planning
  • Take part in sprint planning, challenge it
  • Provide feedback about testing timeline
  • Flag risky estimations.
  • Set  code completion day
  • Set code blitzing day
Product Design  
  • Understand the design needs and its alignment with legacy code
  • Test the design or do walk through with the developer
  • Ask the rationale behind their decision.
  • Ask for prototype completion estimate
  • Validate your testing checklist
Delivery (Local deployment)  
  • Test, test and automate test cases
  • Involve team in testing
  • Create clean bug reports
  • Verify bugs quickly
  • Update testing infrastructure
Staging Deployment  
  • Facilitate UAT
  • Prepare test results report
  • Flag NO GO releases
  • Become a virtual documentation of the product.
  • Stay enthusiastic