Thursday, March 25, 2010

Thinking QA Automation

Introduction
This article will help you get started about thinking from an high level on QA Automation process and QA Automation Framework expectations.

Background
I was asked to attend a QA lead meeting for discussing QA Automation process and framework. I have never been a QA resource but with my previous interactions with few QA teams and browsing thru several articles on net I could get an overall understanding of QA Automation.
As in our case we knew that we will use Selenium tool as it was already being used in some existing projects and as it is open source (and free) and offers several advantages, so I have also listed few findings around Selenium tool.

QA Automation process will involve thinking on following
  1. Capture Business Requirements and maintain traceability between Requirements and test cases to ensure that you have test cases against each requirement.

  2. Write manual test cases against each requirement. Manual test case writing will help as a pseudocode for writing automation tests. Moreover you might not want to write automation tests for functionality which is still not finalized. Also there will be cases (due to limitations of automation testing tools) where you will have to rely on manual testing.

  3. Think about What interfaces does your tooling need to interface with. For instance, do you need to drive a browser, send SOAP/ REST requests and validate its responses, validate email notifications, do database validations, do document or image validations etc.

  4. Choose a tool based on your requirements. There are high end tools available like QTP which are costlier and then there are widely accepted feature rich open source tools also.

  5. Identify tests that can be automated vs manual tests

  6. Grouping test scripts by Module / Use Case

  7. Identify Recovery Scenarios / Execution Conditions when a test script will fail

  8. Integrate Test Automation Scripts in Build process

  9. Choose QA Automation framework. Action Driven / Data Driven / Record & Playback. Data Driven Frameworks are maintainable for mid-large size projects.


Considerations for developing a QA framework include
  1. Error Logging when a script fails. This will indicate a test failure or will indicate a test script change

  2. Notification Rules

  3. Automatic Defect Logging in case of any failure

  4. Storing Test Data in Excel or DB and writing utility functions around retrieving it

  5. How to minimize propagating errors following a failure in one of the scripts

  6. Monitoring of memory consumption, disk space, etc as applicable.

  7. A set coding conventions and best practices for writing test scripts

  8. You can write functions to select a menu choice, pull up a dialog, set a value for a variable, or issue a command. If the UI changes how one of these works, you change how the function works.


Few facts around Selenium
  1. Selenium automation tool is absolutely free. Selenium allows writing and executing test cases in various programming languages including C#, Java, PERL, Python, PHP and even HTML.

  2. Selenium can test Silverlight applications using a extension -http://code.google.com/p/silverlight-selenium/

  3. Selenium weak point is that it can test only web applications. Selenium cannot test Web Services, REST and JSON. Please correct me here if I am wrong.

  4. Selenium supports AJAX testing using waitForxxx functions

  5. Selenium can be integrated with Build process and continuous integration tools like Hudson and Cruise Control

  6. Selenium supports Data Driven Framework


Some Good links that I browsed
  1. Improving the Maintainability of Automated Test Suites - http://www.kaner.com/lawst1.htm

  2. Introduction to Selenium (Good and Bad points): http://www.qaautomation.net/?p=4

  3. This links says that WebTest (another open source tool) is better than Selenium in few things - http://mguillem.wordpress.com/2007/10/29/webtest-vs-selenium-webtest-wins-13-5/

History
Please suggest additions to this article based on your experience and thoughts and I will update the article accordingly.