User acceptance testing: Goals, best practices, and management

In a previous video, you learned about different ways to measure customer satisfaction, including
feedback surveys and user acceptance testing (UAT). This reading will focus on why conducting UAT is
essential to the successful launch of any product, service, or software. We will also discuss some best
practices for effective UAT and how to manage the feedback you receive.

The goals of UAT

To recap, UAT is testing that helps a business make sure that a product, service, or process works for its
users. The main objectives of UAT are to:

 Demonstrate that the product, service, or process is behaving in expected ways in real-world
scenarios.

 Show that the product, service, or process is working as intended.

 Identify issues that need to be addressed before project completion.

UAT simulates real-world conditions, so when the feature works as intended during the testing process,
you can be more confident that your product, service, or process will work properly once it is launched.
It allows a project team to gather detailed information about how users interact with a product, service,
or process. UAT helps the team answer such questions as: Do users recognize its purpose and uses? How
do they interact with it? How much time do users take to interact with it? Do they notice all of its
features? Is the product, service, or process accessible to everyone? UAT also allows the project team to
record information about how users feel about their experience with a product, service, or process.
Through testing, the team can learn about the emotions it evokes, identities it conveys, appeal it holds,
and so on.

Best practices for effective UAT

In order to achieve these goals, UAT needs to be conducted thoughtfully. These best practices can help
you administer effective UAT:

 Define and write down your acceptance criteria. Acceptance criteria are pre-established
standards or requirements that a product, service, or process must meet. Write down these
requirements for each item that you intend to test. For example, if your project is to create a
new employee handbook for your small business, you may set acceptance criteria that the
handbook must be a digital PDF that is accessible on mobile devices and desktop.

 Create the test cases for each item that you are testing. A test case is a sequence of steps and
its expected results. It usually consists of a series of actions that the user can perform to find out
if the product, service, or process behaved the way it was supposed to. Continuing with the
employee handbook example, you could create a test case process in which the user would click
to download the PDF of the handbook on their mobile device or desktop to ensure that they
could access it without issues.

 Select your users carefully. It is important to choose users who will actually be the end users of
the product, service, or process.
  Write the UAT scripts based on user stories. These scripts will be delivered to the users during
the testing process. A user story is an informal, general explanation of a feature written from the
perspective of the end user. In our employee handbook example, a user story might be: As a new
employee, I want to be able to use the handbook to easily locate the vacation policy and share it
with my team via email.

 Communicate with users and let them know what to expect. If you can prepare users ahead of
time, there will be fewer questions, issues, or delays during the testing process.

 Prepare the testing environment for UAT. Ensure that the users have proper credentials and
access, and try out these credentials ahead of time to ensure they work.

 Provide a step-by-step plan to help guide users through the testing process. It will be helpful
for users to have some clear, easy-to-follow instructions that will help focus their attention on
the right places. You can create this plan in a digital document or spreadsheet and share with
them ahead of time.

 Compile notes in a single document and record any issues that are discovered. You can create a
digital spreadsheet or document that corresponds to your plan. It can have designated areas to
track issues for each item that is tested, including the users’ opinions on the severity of each
issue. This will help you prioritize fixes.

Managing UAT feedback

Users provide feedback after performing UAT. This feedback might include positive comments, bug
reports, and change requests. As the project manager, you can address the different types of feedback as
follows:

 Bugs or issues: Users might report technical issues, also known as bugs, or other types of issues
after performing UAT. You can track and monitor these issues in a spreadsheet or equivalent
system and prioritize which issues to fix. For instance, critical issues, such as not being able to
access, download, or search the employee handbook, need to be prioritized over non-critical
issues, such as feedback on the cover art of the handbook.

 Change requests: Sometimes the user might suggest minor changes to the product, service, or
process after UAT. These types of requests or changes should also be managed and prioritized.
Depending on the type and volume of the requests, you may want to share this data with your
primary stakeholders, and you may also need to adjust your project timeline to implement these
new requests.

Common data metrics for project management

There are many types of project data you can use to determine your team’s progress and efficiency,
evaluate the success of your project, and inform project decisions. While you don't need to be a data
expert, knowing how to measure, track, and evaluate the right kind of data will help you deliver the most
value and impact.

This reading will recap some of the common types of data from the previous video and introduce a few
more key data points that can help you manage projects and work with stakeholders. This reading will
also introduce a few ways to interpret the data so that you can reduce risks and make the right decisions
about your teams and projects.

The benefits of analyzing data in project management

As a project manager, you can use data daily to make better decisions, solve problems, improve
performance and processes, and understand your users.

For example, if you have data on customer buying patterns, you can identify your best-selling products,
and you'll be able to make smarter decisions when placing new product orders with your suppliers. This
data will also help you better understand your users and their preferences so you can improve your
product offerings and performance.

You can also use project team data to help you refine your processes. For example, if your team is
experiencing an issue, analyzing data from your project tracker about the number of tasks completed,
escalations, or internal process problems can help you find the source. This will allow you to make an
informed decision about where to focus your efforts to improve processes.

Through critical analysis, application, and execution, data becomes a powerful tool to guide any project
in the right direction.

Data, metrics, and analytics

Three images: Numbers grouped in a circle to represent data; a graph with rulers lining the x and y axis
to represent metrics; two people working on a puzzle to represent analytics.

Data is information. It’s the numbers and feedback available to you about different aspects of your
project. Metrics are how you measure your data. They define the important or specific information
(data) you need to know about your project, such as productivity, quality, or engagement. Once you
determine your project's metrics, you analyze the data according to those metrics to find patterns and
answer questions about your project. This process is called analytics: using data to answer questions,
discover relationships, and predict unknown outcomes.

When analyzing data, ask: What do the metrics mean to you? How do you want to use the metrics
you've chosen? Can you find patterns to make predictions about your project? Can you find ways to
improve—or optimize—certain aspects of your project? What lessons can you draw from your project's
data?

The next few sections are some common categories of metrics used in project management and a brief
explanation of what they are and how they're useful to a project. Keep in mind that your use of different
metrics isn’t limited to these categories. All of your project data is interrelated. The same metric can also
provide different information when applied to different aspects of your project.

Productivity metrics

Productivity metrics typically measure progress and output over time. They allow you to track—or
predict—the effectiveness and efficiency of your project team.

To track your team's productivity over time, analyze the number of tasks or milestones completed in a
given time frame. Ask questions like, what percentage of tasks are completed on time, and how long do
they usually take? Or, if tasks were not completed on time, how much longer than anticipated did it take
to complete all the tasks?

On-time completion rates can help illustrate to clients and stakeholders how the project is progressing
and when they can expect certain deliverables to be ready. If your project's completion rates are high, it
means you're doing a good job of meeting your completion goals. If the rates are low, it means you're
missing deadlines. Analyzing data can help you make decisions about things like improving or
implementing new processes, or re-evaluating how you estimate project scope, complexity, and timeline.

Calculating duration (how long something takes) can be useful for setting and evaluating tasks and
milestones and determining if you'll meet project deadlines. Tracking task duration can improve the
accuracy of estimating a project's timeline. This data is broken down into hours, days, weeks, months,
and sometimes years.

You can also analyze current information to predict future outcomes and make projections (or forecasts)
about productivity trends, project durations, costs, performance or quality. This kind of data empowers
you to proactively manage your project and its resources and measure the accuracy of your projections
over time. For example, analyzing your team's overall performance or velocity can answer questions
such as, is the team completing its tasks and milestones? What percentage of tasks is the team finishing
on time?

Predicting the future may be impossible, but building a better understanding of it and refining your
method for making projections is achievable and valuable.

Quality metrics

Quality metrics relate to achieving acceptable outcomes and can include metrics such as number of
changes, issues, and cost variance, all of which affect quality.

Changes refer to differences in any aspect of the project from what was originally planned or required.
Issues are problems that may affect task completion—and often result in a change. Track the number of
changes and issues to identify patterns, refine processes, and share information about the project with
stakeholders.

Cost or budget variance is the difference between the actual amount of money spent on a project and
the amount that was budgeted for the project. Over time, this data can help you understand how well
you're estimating budgets for your projects. A low variance means you've estimated your project budget
accurately. A high variance means you should reevaluate your estimation process. You could be under- or
over-estimating costs for your budget, or you may not be tracking expenses effectively.

Happiness and satisfaction

Project managers at Google use a sub-set of metrics called happiness metrics that also relate to quality.
These are metrics that relate to different aspects of the user's overall satisfaction with a product or
service, like visual appeal, how likely they are to recommend, and ease of use. Happiness metrics can
generally be captured with a well-designed survey or by tracking revenue generated, customer retention,
or product returns.

Customer satisfaction scores reflect user attitudes, satisfaction, or perceived ease of use. These scores
measure how well the project delivered what it set out to do and how well it satisfies customer and
stakeholder needs. Customer satisfaction scores generally represent a combined metric—the sum of
several different happiness metrics. For example, on a satisfaction survey, a customer might separately
rate a product's appearance as 6/10, ease of use as 7/10, and likeness to recommend or use again as
8/10. The overall customer satisfaction score would then be 7/10.

You will need to determine what scores are acceptable for your project by discussing with stakeholders
what the most important aspects of the project are.

Adoption and engagement

Another set of metrics related to quality are adoption and engagement. Adoption refers to whether or
not a product, service or process is accepted and used. Engagement refers to the degree to which it is
used—the frequency of use, amount of time spent using it, and the range of use. It might help to think of
these in terms of throwing a party: your adoption metrics would reveal to you whether or not people
accepted the invitation and showed up. The engagement metrics would tell you how active they were at
the party—whether they participated in activities or interacted with other attendees, if they invited their
friends to come with them, and how long they stayed.

Adoption metrics for a product or service release, like an app, software program, delivery service, or
gym membership, would be similar to the party example. However, they can be a bit more complex if
you need to track metrics for more than one thing, like whether users make additional purchases or sign
up for premium features.

Each project will need to define its own set of successful adoption metrics, such as:

 Conversion rates
  Time to value (TTV)

 Onboarding completion rates

 Frequency of purchases

 Providing feedback (rating the product or service)

 Completing a profile

Engagement metrics tell you to what degree a product, service, or process is being used. They reveal the
frequency and type of customer interaction and participation over time. Engagement metrics might
include the daily usage rate of a design feature or tracking orders and customer interactions.

As a project manager, you're not only concerned with the end user's level of engagement. It's just as
important to monitor stakeholder and team member engagement as well. Measuring stakeholder
participation by tracking the frequency of communication, responses to emails or updates, attendance
at meetings, or level of input can give you a sense of whether or not stakeholders are finding value in
the project. A lack of meaningful engagement could put your project at risk. Stakeholders may not be
aware of changes or the overall progress of the project, and therefore the final outcome of the project
may not meet their expectations. Measuring team member engagement is vital to the success of your
project because the more engaged they are, the more productive they are, and the more likely they are
to produce high-quality results.

Ideally, you want your adoption and engagement metrics to increase or to at least meet the goal metrics
that were established with stakeholders earlier in the project. If there is no increase, or the metrics
drop, then your rates are low and therefore not as successful. Check out the resources below for a more
in-depth understanding of how and why to measure adoption and engagement.