Fundamentals of

Test Automation
QA & SQA
Quang Le
 Agenda

Introduction An Overview Test Automation Activities Key Success Factors

Course Objective What is Test Automation? Analysis & Design Lessons learned
Learning Method Why do we need Test Automation? Planning How to master test automation
Test Automation Tool Landscape Framework Implementation Summary & Review
Test Script Development
On-going Development
 Objectives

• Gain basic knowledge of concepts, tools, activities,

methodologies in test automation.
• Have awareness of good practices & tips around
test automation to:
 Predict and reduce mistakes
 Increase productivity
 Learning Method

• Actively participate in asking questions

• Do a test at the end of course
An Overview
What is Test Easy to break

Automation? Expensive to Manual Tests

maintain

Slow to run
Workf
low
The use of software Tests Less brittle
(via UI)
or tools to perform Easier maintenance
Acceptance Tests
the checking Faster Feedback
(API/service layer)

Unit Tests/Component Tests

(Programmer Test)
 Greater Test Coverage:
More time for testers doing Doing what manual testing
exploratory/risk-based tests; cannot: Load Test
increase confidence in
quality

Improved Testing
Productivity: Test suites Increased Reusability:
can be run earlier and
nightly; run tests more
often
ROI Test can be run across
different platforms &
environments; consistent
repeatable testing

Earlier time to market:

Quick feedback & Improved motivation:
Reduced test cycle Reduce tediousness, improve
team morale

Why do we need Test Automation?

Test Automation Tool Lan
dscape
Test Automation
Activities
Test Automation • Analysis & Design
1
Activities
2
• Planning

3
• Framework Implementation

4
• Test Script Development

5
• On-going Development
Analysis & Design

Collect Client’s Requirements & Test cases

Draw Automation Mind Maps
Evaluate Tools & Technology

Estimate & Build Technical Proposal

Build Proof of Concept Technical Lead
(QAA/SQA), AM/EM
 Use tool evaluation matrix
Tool Selection  Ask the expert team

Estimation  Capture historical data of your own

 Consider hidden & uncountable cost
Planning

Get Full Set of Test Cases

Draw Test Case & Product Mind Maps
Select Test Cases for Automation

Estimate Test Automation Effort

Build Staffing, Project Plan & Standards (Conventions, Metrics) Technical Lead
Determine Test Deliverables (QAA/SQA), AM/EM,
Tester

 Check testability first

Test Case Selection  Align with client needs

API vs. GUI

 Test Case Selection

SHOULD SHOULD NOT

• Repeatability & reusability • Dealing with images, videos, flash
• Main flows & high risk • Scheduling configuration
function • Checking animation
• Cross Compatibility • Accessing back-end server
• Reliability • A mixture of steps: web + desktop
• Time intensive execution
Framework Implementation

Build Code Base, Structure

Define Coding Standards & Practices
Set up Test Environment & SUT

Automate Test Cases (5% -> 10%)

Technical Lead (QAA),
SDET, Tester/Scripter
 Follow coding standards
Coding Standards &  Select correct patterns
Patterns  Follow practices

 Clean
Traits  Well-structured
 Consistent
Building Test Frameworks
Benefits
 Linear FW
 Test Library Architecture FW
Common Usages  Data-Driven FW • Maintainability: Reduce
 Keyword-Driven FW maintenance cost
 Hybrid Test FW
• Productivity: reuse, run
fast, scalability, etc.
 Target team with coding skill level • Resource Allocation:
Key points
 Time & budget constraints Provide a non-technical
 Framework feature needs
 Current good practices
user interface

 Page Object Model (POM)

Common  Fluent Page Object It Depends on the
Patterns  Singleton
 And so forth Context!
Test Script Development

Learn Framework (keywords, libraries, object identification)

Update & Prioritize Test Cases
Create Test Scripts and Test Suites

Execute Test Scripts & Test Suites

Review Test Scripts, Analyze & Report Test Results
Technical Lead,
Scripter
 F.I.R.S.T Principles
 Prepare your test environment
 Optimizing object identification
Practices  Test Constantly
 Keep up-to-date & review regularly

Goal: Reliable test, less brittle, faster execution & less maintenance
On-going Development

Maintain Implemented Test Framework

Maintain & Add New Test Scripts
Execute, Analyze & Report

Maintain Mind Maps & Guidelines

Monitor & Control (metrics) AM/EM, Technical
Lead, Scripter
 Parallel & Distributed Tests
Practices  CI Integration
 Keep up-to-date & review regularly
 Collaboration
Key Success
Factors
 Lessons learn in Automated
Testing

• Design your tests first, before deciding which to

automate
• Do not mandate 100 percent automation
• Do not automate a mess
• Test automation is a software development process
• Avoid complex logic in your test scripts
• Beware of using automators who don’t understand
testing and don’t respect testing
• Testability is often a better investment than automation.
 How to Master
Test Automation
from Scratch?
Key Steps
Mindset
Hands-on
& Attitude
Knowledge & Skill
• Have testing knowledge
• Basic Test Automation
• Understand HTML, CSS, XPath, Web Development, Browsers
• Learn Add-ons: Firebug, FirePath, XPath Helper, Developer Tool
• Learn a Test Automation Tool – Katalon, UFT, TestComplete
• Learn Java/Python/C# or another programming language
• Learn development environment: Maven, Eclipse, TestNG Test Mindset +
• Learn Selenium in-depth and 6 popular frameworks
• Test Automation Patterns Technical Skill +
• Learn Design Patterns Passion = Test
• Learn other tools, frameworks and related knowledge
• Share and participate in communities and discussions
Automation Expert
• PRACTICE, PRACTICE & PRACTICE
 Takeaways

BASIC CONCEPTS
What? 1
Why? ACTIVITIES
Pyramid Test Analysis & Design

2 Planning
Framework Development
Script Development
PRACTICES
Tool Selection
Test Case Selection 3
Coding Standards
NON-STOP LEARNING
Object Identification
Passion with your career and keep self-
Test Scripts
4 improvement.
Q&A
 Common Test Automation Framework

FUNDAMENTAL TEST AUTOMATION 24

COMMON TEST AUTOMATION FRAMEWORK

Linear FW

Test Library Architecture FW

Data-Driven FW

Keyword-Driven FW

Hybrid Test Automation FW

FUNDAMENTAL TEST AUTOMATION 25

RECORD/PLAYBACK

Popular approach among commercial tool

Capture interaction with system and then replay

FUNDAMENTAL TEST AUTOMATION 26

RECORD/PLAYBACK – PROS & CONS
Pros Cons
1. Fastest and easiest way to create 1. Do not test anything unless
test scripts initially checkpoints added
2. No programming skills needed 2. Reusable limitation
3. Code duplication
4. Require a lot of rework if any
changes
5. Hardcoded data
6. Difficult to manage (project, test
suite, test case, etc.)

FUNDAMENTAL TEST AUTOMATION 27

LINEAR

Steps are written with validations, sometimes use record/playback feature

FUNDAMENTAL TEST AUTOMATION 28

LINEAR – PROS & CONS
Pros Cons
1. Fastest way to create test scripts 1. Reusable limitation
initially 2. Code duplication
2. No programming skills needed 3. Require a lot of rework if any
3. Script Independence – The scripts changes
are independent of each other 4. Hardcoded data
4. Can use common scripting 5. Difficult to manage (test suites, test
languages case, etc.)
6. Scripts are difficult to read
7. Maintaining this code is very time
consuming

FUNDAMENTAL TEST AUTOMATION 29

LIBRARY

Enables creation of library files representing Modules & Functions of SUT

Test Scripts

Reusable actions
Libraries

System Under Test

FUNDAMENTAL TEST AUTOMATION 30

LIBRARY – PROS & CONS
Pros Cons
1. The amount of time spent on coding 1. Spend time to analyze reusable
is lesser: use reusable functions flows
2. Maintenance is easier: make the 2. In some cases, data are still
changes at only one place in reusable hardcoded in the script
function 3. Good programming knowledge is
required to create and maintain
function libraries
4. Spend effort on coding review

FUNDAMENTAL TEST AUTOMATION 31

DATA-DRIVEN

A framework which is driven by test data

Repeated use of Test Scripts with different Inputs and Response Data coming out of predefined Dataset

FUNDAMENTAL TEST AUTOMATION 32

DATA-DRIVEN

FUNDAMENTAL TEST AUTOMATION 33

DATA-DRIVEN – PROS & CONS
Pros Cons
1. Changes to test scripts do not affect 1. More time is needed to plan and
the Test Data prepare both Test Scripts and Test
2. Test cases can be executed with Data
multiple Sets of Data 2. Additional effort and good technical
3. Reducing coding for large test expertise are required to create
cases functions that connect to the external
4. Data inputs or outputs, expected data sources to retrieve the data
result: stored and accessed easily &
conveniently

FUNDAMENTAL TEST AUTOMATION 34

KEYWORD-DRIVEN

Create various keywords and associate a different action or function with each of these keywords. Then, create a
Function Library that contains the logic to read the keywords and call the associated action

FUNDAMENTAL TEST AUTOMATION 35

KEYWORD-DRIVEN

1 2

FUNDAMENTAL TEST AUTOMATION 36

KEYWORD-DRIVEN – PROS & CONS
Pros Cons
1. Able to use for different applications 1. Lot of effort needs to be spent
2. Easy & quick to write the test script initially to create the function libraries
steps in excel sheets (~500 man-hours)
3. Maintenance is low in the long run 2. Need high programming skills to
4. Easy division of Labor create the complex keyword
3. Not easy for new people to
understand the framework quickly
4. Low performance at run time

FUNDAMENTAL TEST AUTOMATION 37

COMMON TAF COMPONENTS
Input Data (XML,
Test Suites Test scripts Data files, Variables,
etc.

SUT
Inside Framework & Tool

User-defined
Built-in Keywords,
Keywords,
Functions
Functions Repository

Test Run, Test Email Notifications, Bug

schedule Test Reports Reports

FUNDAMENTAL TEST AUTOMATION 38

Tool Selection Matrix

Define requirement & expectation

Define & prioritize criteria
Collect list of tools
Allocate suitable resources
Use and Do PoC with samples
Finalize the evaluation
Q&A