Shout out for Automation

Author: Matt Chiang

What is automation:

In the most simplistic term, automation is some type of script that controls an application to do some type of action.  However, there are flaws to this ideology.  The background to automation probably started years ago w/ someone in QA or development when they were doing the same thing over and over again.  They eventually got tired of wasting time so they wrote a script to do the same action for them automatically so they can do something else.  This probably isn’t 100% true but that is how I got started writing automation back 17 years ago.  

Why automation:

Automation can be simple or complex.  The need for automation increases as technology advances.  Things that used to take days, weeks, months to validation will now take minutes or hours.  Machine power is cheap.  Machines never complain about doing the same thing over and over and will always be 100% focused on their task 24 hrs a day/365 days a year.  Machine power makes it cheaper to produce a product.  A look into history will reveal that just about every industry employs some type of automated process.  So why not software.  Testing software is expensive, tedious, monotonous, and error prone if you are not aware of everything that is being tested.  Automation can be built to do the same type of tests over and over and validate every possible point of failure.  Automation is also great at capturing business rules for a functionality that one would have to memorize and exercise.  Most people will end up forgetting about a business rule after a few weeks but automation won’t.  

Progression and need of automation:

In the past, most QA engineers would only need to find some type of macro recorder, or some type of record and play back tool to get started doing automation.  This was great in the days of waterfall development where the application changes were slow to develop.  The recorded script could execute for months before a new one would need to be recorded to test a change in a function.  Nowadays, w/ every company pushing Agile development, changes are frequent and releases are fast.  Code changes daily.  Automation is needed more and more for QA engineers to keep up, especially when it comes time to do regression testing.  Most agile development sprints are 1/2/3 weeks and at the end of the sprint, the code should/needs to be releasable.  How can one test all the new features that was developed and to make sure previous code wasn’t broken because of the change w/o automation?  It’s nearly impossible.

Good and bad automation:

I’ll start w/ bad automation.  In the past, code was development using procedures or basically a really long script of steps for a function to work.  The automation script would also be procedural, following a functional test script until it gets to the end.  This makes for very brittle test scripts and the QA engineer would always have to babysit his tests.  This practice does not work today in the world of object oriented development where developers uses shared libraries/classes to build a function.  This allows developers greater efficiency when building a functionality.  This also creates the need for object oriented automation.  If a QA engineer writes 100 procedural scripts that tests 100 different functional tests and one step happens to change, they would have to change it in 100 different areas.  A good example of this is a simple login.  If there are 100 tests that needs the user to be logged in and the username text field changes it’s value to User Name, then someone would have to change the script.  If the User Name field is procedural in 100 scripts, there 100 locations to change.  If the automation script was written in a object oriented pattern, then only 1 line of the script would have to be change for 100 functional tests to work again.  Good automation is built so that the code is written once and can be applied to multiple situations.  Good automation depends on data to drive the tests, not lines of automation code.  Good automation should be easy to run and can be started by anyone w/o being a QA engineer baby sitting the scripts.  Good automation can be run at anytime of the day or night and give QA, BA, PM, Management or anyone interested in test report indicating every failure or area that needs some attention.  Good automation should not fail w/ every build or simple change that would require someone to change the script w/ every time the test is run.  Good automation should be threaded so multiple tests can be run simultaneously to increase efficiency of regression testing.  Regression tests that would take one QA engineer 100 hours to run can be reduced to 10 hours or less if there are 10 tests that can be run at the same time.  The return on investment in automation would be recognized w/ only a few test runs.  

Do I need manual testing w/ all this automation?

Yes!  Manual testing will always be needed.  However, the time that is spent testing by manual testers are now in the areas that needs attention instead of randomly focusing on some area of the application and missing other critical areas where the code changed.  This gives QA engineers a focused testing task rather than adhoc testing task.  Automation should be looked at as a radar system for QA/Development.  The automation is only going to catch what you want to catch.  It will miss things if the engineer who wrote it missed validation points.  If it’s written correctly, a failed automated test should give the QA engineer an indication of where code was changed, so they can look at why it was changed, and what else needs to be tested.  A failed automation test is not always a bug but could lead to one.  Like the radar system, every blip on the screen is not an enemy target but could lead to one.    

What is missing in automation:

There are so many automation frameworks out there nowadays to help test just about everything that it would probably take a lifetime to master everyone of them along w/ new ones.  So what is missing?  The biggest piece to this automation puzzle is reporting.  Every framework will give the QA engineer a report at the end of each test run.  However, it only applies to that one time.  What if someone wants to look at historical trends?  Other than manually tracking down a test report or log, there isn’t much that is available.  However, great QA engineers will come up w/ great solutions.  Like developers building an application for customers, QA engineers builds solutions to capture test results.  A good reporting application will make great automation awesome.  It takes away the complexity from various group members in interpreting the reports or logs.  The reporting application should be accessible to anyone one the team or management.  It will give them 24/7 access to the test results and ways to query for certain tests or to look at failures.  It gives them step by step analysis of where the test failed and a screenshot of what the application looked like when it did fail.  W/ the reporting application, automation tests can run 24/7, 365 days a year.  Failed tests will need to be analyzed manually but the QA engineer does not need to babysit the tests anymore.  

Final thoughts:

Just as anything in technology, the implementation has to be correct.  Poor implementation of a technology w/ only lead to frustrations, re-factoring of code, or complete re-writes.  The person doing the work needs to know what they are doing w/ the tool.  However, w/ the correct implementation of automation, applications can be released more frequently that is more stable and reliable.  The last thing anyone wants, is to have the customers test the application because QA didn’t have enough time to test it.  
Author: Matt Chiang

Posted in QA