By: Bill Witt
Introduction
As a Quality Assurance Engineer or Software Development Engineer in Test, there is high demand for those who are not only able to create automated tests using Selenium WebDriver, but there is also a demand for engineers who can integrate selenium testing into a continuous integration server like Jenkins.
Using tools like Jenkins provides the ability to create and maintain automated test runs and provide test result reports.  Mastering Jenkins can be a challenge to those who may be unfamiliar with it and its many configuration options and plugins, but the challenges presented by Jenkins can be even more discouraging when trying to set it up for a Linux environment.
Problem Background
Having been familiar with configuring WebDriver automation tests to run on an instance of Jenkins from within a Windows environment, I had hoped that configuring Jenkins might be as straight-forward when configured in a Linux environment, in this case, Ubuntu.  Upon beginning this endeavor, I quickly found that setting up a Linux based Jenkins configuration was quite a bit more difficult, especially in a non-GUI Linux Server environment.  
The following article has been written with the intention of assisting others who may have come across the same difficulties I had encountered, or those who may have been tasked with setting up a Jenkins automated test project who may not be sure where to begin.
Solution
After several days of trial and error attempting to run automated Firefox and Chrome WebDriver tests from Jenkins in a Linux (Ubuntu) environment, I was finally able to implement a solution that works.  This solution will work in Linux environments that do and do not have a GUI, both desktop and server versions.
In the following sections of this article I will outline the various steps required to successfully configure the Linux environment and Continuous Integration Server, in this case Jenkins, to run WebDriver automated tests using either the Firefox or Chrome browsers.
Linux Setup
The first thing to do is to prepare your Linux environment by adding the required Chrome and Firefox drivers.  Although somewhat obvious that chromedriver will need to be setup separately, due to recent changes in how WebDriver works, it will also be necessary to separately configure Firefox’s driver, geckodriver.  Run the following commands in the Linux terminal, outlined in each of the driver configuration sections.
Chromedriver Setup:

1. 1. sudo apt-get -y install unzip

1. 1. wget -N http://chromedriver.storage.googleapis.com/2.10/chromedriver_linux64.zip -P ~/Downloads

1. 1. unzip ~/Downloads/chromedriver_linux64.zip -d ~/Downloads

1. 1. chmod 777 ~/Downloads/chromedriver

1. 1. sudo mv -f ~/Downloads/chromedriver /usr/bin/chromedriver

1. sudo apt-get install libxi6 libgconf-2-4

Geckodriver (Firefox) Setup:

1. 1. sudo apt-get -y install unzip (skip this step if done previously)

1. 1. wget -N https://github.com/mozilla/geckodriver/releases/download/v0.16.1/geckodriver-v0.16.1-linux64.tar.gz -P ~/Downloads

1. 1. tar -xvzf ~/Downloads/geckodriver* -C ~/Downloads

1. 1. unzip ~/Downloads/geckodriver* -d ~/Downloads

1. 1. chmod 777 ~/Downloads/geckodriver

1. sudo mv -f ~/Downloads/geckodriver /usr/bin/geckodriver

Xvfb Setup:

1. 1. sudo apt-get -y install xvfb

1. sudo chmod 777 /run/user/

Git Setup:

1. sudo apt-get -y install git-all

Jenkins Setup
After completing the installation of the Chrome and Gecko (Firefox) drivers, you will need to configure Jenkins to properly run the WebDriver tests in a semi-headless browser type of execution.  The following configuration sections will provide the steps for installing the required Jenkins plugins and Jenkins build environment setup.  This article assumes that Jenkins has already been installed, but if this is not the case, an additional section has been added to the end of this article that provides the steps needed to install Jenkins
Jenkins Plugin Installations:

1. 1. Open the plugin manager by doing the following, navigate to: Jenkins > Manage Jenkins > Manage Plugins, then click the “Available” tab

1. Select the following plugins by selecting from the alphabetized list, or by typing the plugin name in the “Filter” textbox, then click the associated checkbox and click the “Install without restart” button (repeat for each plugin)
   1. 1. GitHub Organization Folder
   1. 1. Pipeline
   1. 1. Xvfb
   1. Safe Restart (this is optional, but recommended)

Jenkins Global Tool Configuration:

1. 1. Open the Jenkins Global Tool Configuration window by navigating to: Jenkins > Manage Jenkins > Global Tool Configuration

1. 1. Scroll to the “JDK” section and do the following:
      1. 1. Click the “Add JDK” button
      1. 1. For the name textbox, just enter: JDK 8 (this is whatever Java version you happen to be using, this could be 7, 8, 9, etc.)
      1. 1. Click the “Install automatically” checkbox
      1. 1. Select the “Install from java.sun.com” option, then select the appropriate version
      1. Click the “I agree…” checkbox. You may have to enter the Java/Oracle username credentials, which can be setup for free on their website

1. 1. Scroll to the “Git” section and do the following:
      1. 1. Click the “Add Git” button
      1. 1. Name the Git Installation Default or Git
      1. In the “Path to Git executable” textbox, enter: /usr/bin/git

1. 1. Scroll to the “Maven” section and do the following:
      1. 1. Click the “Add Maven” button
      1. 1. Just name the installation Maven or Maven <version number>
      1. 1. Click the “Install automatically” checkbox
      1. Select the version installed (usually the latest version available) in the dropdown box under “Install form Apache”

1. 1. Scroll to the “Xvfb installation” section and do the following:
      1. 1. Click the “Add Xvfb installation” button
      1. 1. Name the installation simply, Xvfb
      1. Click the checkbox by “Install automatically”

1. Click the “Save” button

Jenkins System Configuration:

1. 1. Open the Jenkins System Configuration by navigating to: Jenkins > Manage Jenkins > Configure System

1. 1. Scroll to the “Global properties” section

1. 1. Click the checkbox by “Environment variables”

1. 1. Add a new environment variable by clicking the “Add” button

1. 1. Name the variable: XDG_RUNTIME_DIR

1. 1. For the value, enter: /run/usr/1001

1. 1. Scroll to the “Jenkins Location” section

1. 1. In the “Jenkins URL” textbox, enter: http://127.0.0.1:8081

1. Click the “Save” button

Create the Job for the Automation Project:

1. To create a new project for your automated tests, do the following:
   1. 1. From the main Jenkins vertical menu, select: “New Item”
   1. 1. Name the project in the “Enter an item name” textbox
   1. 1. Select the “Freestyle project” option
   1. Click the “OK” button

Configure Jenkins Automated Test Project/Job:

1. 1. Select the newly created project, then click “Configure” in the left vertical menu

1. 1. Scroll to the “Source Code Management” section
      1. 1. Click the “Git” radio button
      1. 1. Set the repository URL in the associate textbox (e.g., https://github.com/<username>/<projectname>)
      1. 1. If the GitHub repo is Private, rather than Public, add the necessary credentials
      1. 1. Scroll to the “Branches to build” section
      1. In the “Branch Specifier” textbox enter: */master

1. 1. Scroll to the “Build Environment” section
      1. 1. Click the checkbox by the “Start Xvfb before the build, and shut it down after.” Option
      1. 1. Click the “Advanced…” button
      1. Select the Xvfb installation previously created from the drop down menu

1. 1. Scroll to the “Build” section
      1. 1. Click the “Add build step” button
      1. 1. Select the “Invoke top-level Maven targets” option
      1. 1. Select the previously named Maven installation from the dropdown menu
      1. 1. In the “Goals” textbox enter: clean test
      1. 1. Click the “Advanced…” button
      1. In the “POM” textbox enter: $workspace/pom.xml

1. 1. Scroll to the “Post-build Actions” section
      1. 1. Click the “Add post-build action” button
      1. 1. Select the “Publish TestNG Results” option
      1. In the “TestNG XML report pattern” textbox enter: **/testng-results.xml

1. Click the “Save” button

Jenkins Linux Installation
In the case that Jenkins has not yet been installed on whichever Linux environment you may be setting up your tests in (preferably in Ubuntu), these are the steps for installing the Jenkins continuous integration server in your environment, run from the terminal window:

1. 1. wget -q -O – https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add –

1. 1. sudo sh -c ‘echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list’

1. 1. sudo apt-get update

1. 1. sudo apt-get install jenkins

1. Edit the /etc/default/jenkins config file to replace the line, “HTTP_PORT=8080” with “HTTP_PORT=8081”

Sample WebDriver Setup for Automation Project
Initializing the Java WebDriver instance for Chrome or Firefox in your project can be accomplished by doing the following:
public static String chromeDriverPath = “/usr/bin/chromedriver”;
System.setProperty(“webdriver.chrome.driver”, chromeDriverPath);
WebDriver driver = new ChromeDriver();
public static String geckoDriverPath = “/usr/bin/geckodriver”;
System.setProperty(“webdriver.gecko.driver”, geckoDriverPath);
WebDriver driver = new FirefoxDriver();
Additional Email Configuration Options
Another essential set of features that should be configured in your Jenkins setup includes collecting test result data into a test report, displaying test result data, and enabling the ability for Jenkins to email the test result report.
TestNG Emailable Report Setup
Before test reports can be sent out after a test run, the settings for both the native Jenkins email publisher and the “Email Extension Plugin” will need to be configured.  This configuration will require that the SMTP Server setting be setup from within Jenkins in two separate locations.
Once the Email notification settings have been configured, the next step will be to install and configure the “TestNG Results Plugin” and optionally, the “Test Results Analyzer Plugin.”  The TestNG Results Plugin will compile all the test run and result data into an emailable report that can be automatically sent off once a test fun has completed.  The Test Results Analyzer Plugin is not required, but it adds an additional presentation of the test result data.
Install Plugins

1. 1. Open the plugin manager by doing the following, navigate to: Jenkins > Manage Jenkins > Manage Plugins, then click the “Available” tab

1. Select the following plugins by selecting from the alphabetized list, or by typing the plugin name in the “Filter” textbox, then click the associated checkbox and click the “Install without restart” button (repeat for each plugin)
   1. 1. Email Extension Plugin
   1. 1. Test Results Analyzer
   1. TestNG Results

System Email Notification Configuration

1. 1. Open the Jenkins System Configuration by navigating to: Jenkins > Manage Jenkins > Configure System

1. 1. Scroll to the “Extended E-mail Notification” section

1. 1. Click the “Advanced…” button

1. 1. Fill in the SMTP server details as follows (using the STG Consulting Gmail as an example):
      1. 1. In the “SMTP server” textbox, enter: smtp.gmail.com
      1. 1. In the “Default user E-mail suffix” textbox, enter: @stgconsulting.com
      1. 1. In the Advanced section, check the box by “Use SMTP Authentication”
      1. 1. Enter the STG Gmail username (email address)
      1. 1. Enter the STG Gmail password
      1. 1. Click the “Use SSL” checkbox
      1. 1. In the “SMTP port” textbox, enter: 465
      1. Change the Default Content Type option to “HTML (text/html)”

1. 1. Scroll to the “E-mail Notification” section

1. 1. Click the “Advanced…” button

1. 1. Fill in the SMTP server details with the same information that was added previously in step 4

1. Click the “Save” button

TestNG Results Configuration

1. 1. Navigate to the Job/Project page for the test automation

1. 1. Select the “Configure” option

1. 1. Click the “Post-build Actions” tab at the top, or scroll down to that section

1. 1. Configure the TestNG Results Plugin by doing the following:
      1. 1. Click the “Add post-build action” button
      1. 1. Select the “Publish TestNG Results” option
      1. 1. In the “TestNG XML report pattern” textbox enter: **/testng-results.xml
      1. For additional, optional TestNG Results options, click the “Advanced…” button

1. 1. Configure the Email Notifications Plugin by doing the following:
      1. 1. Click the “Add post-build action” button again
      1. 1. Select the “Editable Email Notification” option
      1. 1. Change the Content Type option to “HTML (text/html)”
      1. 1. Click the “Advanced Settings…” button
      1. 1. In the “Triggers” section, in the “Failures – Any” box, do the following
            1. 1. Click the red ‘X’ on the “Developer’s” box to remove it
            1. 1. Click the “Add” button in the “Send To” portion of the box
            1. 1. Select the “Recipients List” option
            1. 1. Click the “Advanced…” button
            1. 1. In the “Recipient List” textbox, enter the email(s) that the test reports will be sent to
            1. 1. If desired, a reply-to email address may be entered in the “Reply-To List” textbox
            1. 1. Change the Content Type option to “HTML (text/html)”
            1. In the “Attachments” textbox, enter: **/emailable-report.html, **/Run Tests.html, or **/*.html
      1. 1. Click the “Add Trigger” button
      1. 1. Select the “Success” trigger option and do the following
      1. Follow the same sub-steps provided in step 5e for the “Success” trigger

1. Click the “Save” button

Test Results Analyzer Configuration
Once the Test Results Analyzer Plugin has been installed, most of its functionality will be automatic.  On the automation project’s page, a new menu item will be added under the new “TestNG Results” option, named “Test Results Analyzer” in the left side menu.  Additional test result analysis can be found by clicking this menu option.  For further configuration, do the following:

1. 1. Open the Jenkins System Configuration by navigating to: Jenkins > Manage Jenkins > Configure System

1. 1. Scroll down to the “Test Results Analyzer” section

1. Configure the settings as desired within this section, such as specifying the graph and chart types and thresholds

I hope these steps were helpful to better understand how to successfully configure the Linux environment and a Continuous Integration Server, like Jenkins, to run automated tests using Selenium webdriver on either Firefox or Chrome browsers.
For more information regarding selenium testing, jenkins integration, or automated testing check out our other blog posts.
To learn how you can work with a STG Consultant for your software technology needs, contact us— we’d love to hear from you.

Author: Matt Chiang

What is automation:

In the most simplistic term, automation is some type of script that controls an application to do some type of action.  However, there are flaws to this ideology.  The background to automation probably started years ago w/ someone in QA or development when they were doing the same thing over and over again.  They eventually got tired of wasting time so they wrote a script to do the same action for them automatically so they can do something else.  This probably isn’t 100% true but that is how I got started writing automation back 17 years ago.  
 

Why automation:

Automation can be simple or complex.  The need for automation increases as technology advances.  Things that used to take days, weeks, months to validation will now take minutes or hours.  Machine power is cheap.  Machines never complain about doing the same thing over and over and will always be 100% focused on their task 24 hrs a day/365 days a year.  Machine power makes it cheaper to produce a product.  A look into history will reveal that just about every industry employs some type of automated process.  So why not software.  Testing software is expensive, tedious, monotonous, and error prone if you are not aware of everything that is being tested.  Automation can be built to do the same type of tests over and over and validate every possible point of failure.  Automation is also great at capturing business rules for a functionality that one would have to memorize and exercise.  Most people will end up forgetting about a business rule after a few weeks but automation won’t.  
 

Progression and need of automation:

In the past, most QA engineers would only need to find some type of macro recorder, or some type of record and play back tool to get started doing automation.  This was great in the days of waterfall development where the application changes were slow to develop.  The recorded script could execute for months before a new one would need to be recorded to test a change in a function.  Nowadays, w/ every company pushing Agile development, changes are frequent and releases are fast.  Code changes daily.  Automation is needed more and more for QA engineers to keep up, especially when it comes time to do regression testing.  Most agile development sprints are 1/2/3 weeks and at the end of the sprint, the code should/needs to be releasable.  How can one test all the new features that was developed and to make sure previous code wasn’t broken because of the change w/o automation?  It’s nearly impossible.
 

Good and bad automation:

I’ll start w/ bad automation.  In the past, code was development using procedures or basically a really long script of steps for a function to work.  The automation script would also be procedural, following a functional test script until it gets to the end.  This makes for very brittle test scripts and the QA engineer would always have to babysit his tests.  This practice does not work today in the world of object oriented development where developers uses shared libraries/classes to build a function.  This allows developers greater efficiency when building a functionality.  This also creates the need for object oriented automation.  If a QA engineer writes 100 procedural scripts that tests 100 different functional tests and one step happens to change, they would have to change it in 100 different areas.  A good example of this is a simple login.  If there are 100 tests that needs the user to be logged in and the username text field changes it’s value to User Name, then someone would have to change the script.  If the User Name field is procedural in 100 scripts, there 100 locations to change.  If the automation script was written in a object oriented pattern, then only 1 line of the script would have to be change for 100 functional tests to work again.  Good automation is built so that the code is written once and can be applied to multiple situations.  Good automation depends on data to drive the tests, not lines of automation code.  Good automation should be easy to run and can be started by anyone w/o being a QA engineer baby sitting the scripts.  Good automation can be run at anytime of the day or night and give QA, BA, PM, Management or anyone interested in test report indicating every failure or area that needs some attention.  Good automation should not fail w/ every build or simple change that would require someone to change the script w/ every time the test is run.  Good automation should be threaded so multiple tests can be run simultaneously to increase efficiency of regression testing.  Regression tests that would take one QA engineer 100 hours to run can be reduced to 10 hours or less if there are 10 tests that can be run at the same time.  The return on investment in automation would be recognized w/ only a few test runs.  
 

Do I need manual testing w/ all this automation?

Yes!  Manual testing will always be needed.  However, the time that is spent testing by manual testers are now in the areas that needs attention instead of randomly focusing on some area of the application and missing other critical areas where the code changed.  This gives QA engineers a focused testing task rather than adhoc testing task.  Automation should be looked at as a radar system for QA/Development.  The automation is only going to catch what you want to catch.  It will miss things if the engineer who wrote it missed validation points.  If it’s written correctly, a failed automated test should give the QA engineer an indication of where code was changed, so they can look at why it was changed, and what else needs to be tested.  A failed automation test is not always a bug but could lead to one.  Like the radar system, every blip on the screen is not an enemy target but could lead to one.    
 

What is missing in automation:

There are so many automation frameworks out there nowadays to help test just about everything that it would probably take a lifetime to master everyone of them along w/ new ones.  So what is missing?  The biggest piece to this automation puzzle is reporting.  Every framework will give the QA engineer a report at the end of each test run.  However, it only applies to that one time.  What if someone wants to look at historical trends?  Other than manually tracking down a test report or log, there isn’t much that is available.  However, great QA engineers will come up w/ great solutions.  Like developers building an application for customers, QA engineers builds solutions to capture test results.  A good reporting application will make great automation awesome.  It takes away the complexity from various group members in interpreting the reports or logs.  The reporting application should be accessible to anyone one the team or management.  It will give them 24/7 access to the test results and ways to query for certain tests or to look at failures.  It gives them step by step analysis of where the test failed and a screenshot of what the application looked like when it did fail.  W/ the reporting application, automation tests can run 24/7, 365 days a year.  Failed tests will need to be analyzed manually but the QA engineer does not need to babysit the tests anymore.  
 

Final thoughts:

Just as anything in technology, the implementation has to be correct.  Poor implementation of a technology w/ only lead to frustrations, re-factoring of code, or complete re-writes.  The person doing the work needs to know what they are doing w/ the tool.  However, w/ the correct implementation of automation, applications can be released more frequently that is more stable and reliable.  The last thing anyone wants, is to have the customers test the application because QA didn’t have enough time to test it.  
Author: Matt Chiang