24th International Conference on Production Research

A PERFORMANCE MEASUREMENT SYSTEMS DESIGN FRAMEWORK

R. Duarte, F. Deschamps, E. Pinheiro de Lima

Industrial and Systems Engineering Graduate Program (PPGEPS),
Pontifical Catholic University of Paraná (PUCPR),
Curitiba, Parana, Brazil

Abstract
Performance Measurement Systems (PMS) are becoming ubiquitous in institutions for several reasons. PMS
are being used to support actions in some day-to-day situations such as decision-making, communicating
expectations, taking preventive measures and learning and understanding the processes, business
environment and markets in which a company operates. There are a few reasons that lead to systems remain
unchanged for an extended period, and often it requires the work of experts to design, analyze and implement
a PMS. This paper presents a framework that, through a structured set of processes, guides the development
of a PMS. This framework does not target companies of specific industries or sizes and is intended to be used
to assess, design and implement a PMS for organizational units, such as departments, or even stand-alone
initiatives, like projects.

Keywords:
Performance Management System; Design Framework; PMSDF

1 INTRODUCTION that guides the professional through the stages of the

The interest on Performance Measurement Systems PMSDF.
(PMS) is not new. However, it’s application is still 2 PERFORMANCE MEASUREMENT SYSTEMS
increasing by companies of the most diverse areas and DESIGN FRAMEWORK
sizes, and for different reasons. PMS may have a great This research builds on previous PMS design frameworks
impact on operations, being applied to measure [6], [5] and research [1]. The main reference model was
performance, manage the strategy, communication, the process described by Neely et al. [5], presented in
influence behavior, learning, and improvement [1]. Figure 1. On this process, the development of a PMS is
To have positive impacts, these applications must be guided by a detailed set of tasks, to assess the company’s
aligned to the company strategy, offering the needed data strategy and market position, and the production
to pursue this strategy to the managers [2]. processes.
Several process models for the development of PMS are Based on this process, the objective was to create a more
presented in the literature. Kaplan and Norton’s balanced simplified process, presenting lower complexity
scorecard (BSC) [3] suggests that the company measures procedures, followed by easy to follow instructions for the
must answer to 4 perspectives: customer, internal, practitioner in charge of the PMS project. Although
innovation and learning and financial. Bourne et al. [4] simplified, the framework is in line with contemporary
propose a model where the development is divided in performance measurement (CPM), linking the measures to
design, implementation, and use of the performance the organization's business strategy [7].
measures. Neely et al. [5] developed a process in which The Performance Measurement Systems Design
the framework presented in the current paper is mainly Framework (PMSDF) (Figure 2) is composed of 5 main
based in which the measures and system are results of stages:
the business strategy.
1. Company Identification
Enterprises are complex systems, and to properly develop
the PMS it is necessary to have a broader view of its 2. Competitive Diagnosis
internal and external relations. These systems integrate 3. Operations Strategy
business process and technologies and must make 4. Business Process Modeling
accessible to the users the information in a clear and easy
5. PMS Processes and Performance Measures
way. This way, the project, deployment, and review of
these systems usually involve skilled professionals This framework was first used in a systematic literature
working with complex factors and technologies, which review (SLR) article about urban mobility indicators [8].
increases the required resources. Aiming to identify if the selected set of papers presented
sufficient elements to understand the PMS in the studied
In some cases, the cost of conducting a PMS
context, it was identified elements related to all the steps
implementation project is high. Operations of medium
of the PMSDF, indicating that the article's sample fitted to
sized companies, not-for-profit institutions, autonomous
the research objective.
operations, standalone projects, to list a few examples, are
cases where the corporation operates on a tight budget, The following subsections will discuss the stages and
and to have a consulting company set up or update a PMS activities of the framework and its forms.
is an unrealistic excessive cost. On the other hand, these
2.1 Company Identification (A1)
are cases where a simpler option could fit their needs.
Description: In this session, the company and the team
In this paper, it will be discussed a framework that
involved in the processes of analysis, design, and
simplifies the process of development of a PMS – PMSDF
implementation of the PMS are identified.
(Performance Management System Design Framework). It
will also be presented two application cases where the Objective: Initial identification of the business and the
framework was used in the form of a form-based process individuals with roles in the project and definition of a
schedule of activities.
 Strategy & Performance
Market Based Resource Based

Awareness
1 Product grouping What does success look
like?

2 Business objectives Why Measure Insight

Where to focus?

Whats are the resources

3 Strategy charting 3 performance measures

How Important are they?

4 Can current strategy achieve 4 Signing off measures
our objectives Development

5 Navigating towards our 5 Review measures Measurement of

objectives competences

Figure 1: Strategy and Performance [5]

Company Target Audience Segmentation

Basic information, the area of activity, size, and annual Assessment of the target audience segmentation identified
revenues. and serviced by the company.
Contacts Business success critical factors
Contact options (telephone and e-mail) and the role of the Evaluation of adopted strategy characteristics: product
people who support the project. characteristics, production, quality, flexibility, and pricing
policies.
Interview Schedule
Initial task definition for upcoming meetings. Positioning related to competitors
Identification of the main direct competitors of the
2.2 Competitive Diagnosis (A2) company, the market leader and the indicators used for
Description: In this stage, it is identified how the objectives comparison.
of the company are currently defined, and if they are
coherent with the business identity. Strategic objectives
Objective: To verify how the company is performing in the Evaluation of the company's defined strategic objectives.
market in which it is inserted, compared to the main SWOT
competitors and the interests of its customers. Another
goal here is to identify the culture and strategy of the SWOT analysis, displaying Strengths, Weaknesses,
institution and the market to identify what is considered Opportunities, and Threats of the assessed business or
performance in this context. During this stage, it is Operation.
identified the depth of the analysis, indicating if the PMS 2.3 Operations Strategy (A3)
will assess a project, a department, an operation or an
Description: The objective of this stage, is to identify the
organizational unit, per example.
characteristics of the company's operations strategy,
Method: Company's representatives must provide the categorizing the performance dimensions by performance
business information, through, for example, interviews, orientations, and comparing these
questionnaires or workshops. Market information may also
orientations to what the competitors practice and what the
be obtained from this internal source, but when available
market expects.
can be obtained from external data, such as market
research. Objective: To verify, among the presented performance
orientations, which need to be redefined by management,
Organizational Mission due to lack or excess of efforts.
Assessment of the company's mission statement and its Method: Data can be obtained from the company
effectiveness. representatives or through customer’s opinion surveys and
Organizational View market research.
Assessment of the company’s vision statement and its Performance Dimensions
effectiveness. List the performance orientations [9], critical to the
Company Values business, and the corresponding performance dimensions,
also describing these dimensions. In this session, evaluate
Assessment of the company's values statement and its the dimensions according to the values in the manufacture
effectiveness. or service tables.
Target Audience Importance-Performance
Assessment of the target audience defined and served by Categorize the performance dimensions listed in the
the company. previous session (Performance Dimensions) regarding
their (A) importance to customers, and (B) performance
 • Company information
• Pre Project Company • PMS Project
Provided Identification Stakeholders
Information • Interviews Schedule
A1 • Mission / Vision / Values
• Approach level • Target audience
• Market segmentation
• Success factors
Competitive • Market positioning
• In situ observation Diagnosis • Strategic objetives
• Interview • SWOT
• Workshop A2 • Performance Dimension
• Document analysis (Manufacture / Services)
Operations • Importance-Performance Analysis
Strategy • Decision areas
• Performance Objectives
A3 • Processes models (BPM)
• Processes performance
objectives
Business Process
Modeling
• Market research
• Customers survey A4
PMS Processes
and Performance
Controls Measures
(restrictions) • Design Factors • Order winning and
A5
order qualifying criteria
• Project team
• Indicators modeling
Step
Inputs Outputs • Deployment schedule
• Project budget / disks /

Figure 2: Performance Measurement Systems Design Framework (PMSDF)

Index quality control
• Indicators deployment
Mechanisms • Action plan
(resources) IDEF0 Notation • Indicators review
24th International Conference on Production Research
 1

Better than
2 Excess Appropriated

Performance Against Competitor

3

4 2
ity
ptabil
Same as 5 nd of Acce
Bou
Lower
6

7 3

Improvement
Worse than

9 1 Urgent Action

10
10 9 8 7 6 5 4 3 2 1
Less important Qualifying Order Winning

Importance for Customer

Figure 3: Importance-Performance Matrix

compared to the competition. Distribute the tabulated recommended that company stakeholders participate
values in the Importance-Performance Matrix (Figure 3) during the choice of indicators and measures, as well as in
[10]. the evaluation and review of the process.
Decision Areas Order winners and qualifiers
Identify current practices among the decision areas of the Based on the identified request-winning criteria (from the
company and what performance dimensions these Importance-Performance Matrix), and after verifying the
practices address [9]. current or necessary indicators in the analyzed processes,
relate indicators and processes, criticality classifying them.
Performance Objectives
Identify, in the Importance-Performance matrix, the Project team
performance orientations positioned the most distant from Define who is responsible for the tasks of the subsequent
the appropriate performance zone, for lack or exceeding phases of the implementation/review process of the
performance, and create a table by orientation. Check with performance measurement system. Some responsible
which decision areas each performance dimension relates. listed in the contacts list may appear at this stage.
Identify weaknesses in decision areas or what forces Modeling the indicators
needed to improve performance within the decision area, Create the indicator sheet [6], for the selected indicators
and what possible strategic actions or choices should be presented in Order Winners and Qualifiers, list If the
made to address those weaknesses or develop strengths. implementation will not work on all indicators or if it was
2.4 Business Process Modeling (A4) decided a scaled deployment, it is recommended to use
the “criticality” scale to prioritize which ones to model first.
Description: Identification of key customer value
development processes related to identified performance Implementation schedule
objectives. Create a list of activities to be developed for the definition,
Objectives: To know the production processes to deliver testing, review, and implementation of the performance
products or services to customers, and the respective measurement system. It is suggested to use Gantt Chart.
measurements and measures.
Budget
Method: Process information can be obtained from
interviews with people involved, with managers, by Identify the direct and indirect costs related to the project,
process documentation or by researcher immersion. the payment deadlines, and the creditors.
It is advised to create models of the processes using Possible costs relate to the acquisition of technology for
graphical notations, like BPMN (Figure 4) or Flowchart, the measurement, storage, and processing of data, hiring
and identify the indicators from the tasks, when possible. project or team consulting to deal with the increased
workload for collection and management of information.
2.5 Performance Measures (A5)
Quality Control
Description: Indicators identification and implementation
Actions to control the performance measurement process
design.
activities. It is related to the methods and instruments used
Objectives: Select, model, test and define the process of in the collection and analysis of data. Control methods
implementation, evaluation, and review of the indicators. may be applied in the context of team training, evaluation
Distribute the activities by the team responsible for the of the measurements responding to the questioning about
process. the analyzed process, whether the frequency or size of the
Method: The selection of the indicators is based on the sample is adequate, if the accuracy adopted is appropriate
information gathered in the previous stage. It is to the needs, to mention a few.
 24th International Conference on Production Research

Order Request

Start Middle End

Confirmed orders rate

Commercial

Production order
Schedule delivery
confirmation

Customer call Default rate

Lead time (suppliers)

Purshasing /
Financial

Order materials Receive payment

from suppliers
Production /
Logistics

Deliver the order

Make product

Orders timely delivered

Figure 4: Example of process mapped in BPMN

Risks of Implementation Performance Measurement System Review

The risks are related to the uncertainty of variables and This phase presents which changes (triggers) occurred in
the possibility of project failure. Examples of risk are that the company impact the defined indicators. An increase in
the indicator is not relevant to the problem to be corrected, the quality requirement may reduce the range of measures
or where the deviation expected to be eliminated is defined (5.10), requiring a new action plan (5.11). The
identified as inherent to the process studied or the cost to review can be initiated by some event or occurs
reduce it does not reward. Identifying the probability of risk periodically.
varies according to the process analyzed, but can be 3 APPLICATION CASES
estimated from company or market history, or with the help
of experts, workshops, panel data, among others. Based on the framework, it was created a Microsoft Word
Macro-Enable Template Document (.DOTM file), with
Testing the implementation of indicators resources like forms fields, macros, and Microsoft Visio
It is recommended that the test be done with a reduced diagrams, with the intention of facilitating the execution of
sample, reducing the risk of impact on the production. the process, not depending on expensive and specialized
When possible, it should be performed using of low-cost ICT platforms.
instruments such as leased tools, external servers, The PMSDF was applied in two cases. The first case was
cheaper equipment with limited accuracy, to assess on sales and production operations of a medium sized
whether the solution designed answers the questions that upholstery factory. In this company, data was obtained
the measurement proposes. Evaluate if the proposed from the company’s representatives on structured
solution is adequate and, if not, why not and what interviews and confirmed during scheduled visits, twice a
measures must be taken for correction. month, through four months. It was verified that there was
no clear market segmentation nor a steady production
Indicators Implementation strategy. By the customer perspective, no major problems
After the tests, this section will manage the actual identified in the market positioning, since the company
deployment in the production environment. Present which was the reference in its market. The collected data was
indicators are being modeled and the progress. not supporting the decision making, and in many
opportunities, the printed production orders and the digital
Indicators Measurement
ones were not with the same information, due to the
For each measured indicator, display its target and the inconsistent update process. It was identified productivity
lower and upper limits. Targets can be objective (weight, losses due to internal processes. No reports were taken,
temperature, quantity, frequency) or relative (standard and no strategic planning existed. By following the
deviation, maximum, minimum, percentage, quartile). framework steps, the company defined a better a set of
Action plan mission, vision, and values, sharing it with the production
team. The process was modified, and the service orders
Check the unwanted results in the Indicators Measurement were then updated in the system daily. Based on the
section and propose action plans for adequacy. The plan information obtained by the collected data, changes were
can be related to the production process or the made in the quality control, the truck loading, and the
measurement process. The action plan may include product delivery processes. Such changes reduced the
changes in the production process or the performance losses by logistics, rework and unnecessary material
measurement process, such as changes in criticality or purchase.
characteristics of indicators.
 Table 1: PMSDF Application Cases
Upholstery Factory Electric car Operation
Company identification Company’s representative Operation Manager
Competitive Diagnosis No definition of market segmentation; No definition of the operation’s objective;
No definition of mission/vision/value Proposed a performance objective based on
statements companies’ statements
Operation Strategy Most performance objectives on No target audience defined
appropriate area
Business Process Identified high quantity of quality issues Operation not aligned to a strategic objective
Modeling Delivery delay by supplier
PMS Processes and Started updating orders’ information in the Several indicators with little or no utility;
Performance system Suggested a set of urban mobility performance
Measures indicators and related objectives

In the second case, the objective was to evaluate a joint 6 REFERENCES

operation in the electric car sector, involving five large [1] Franco-Santos M., Kennerley M., Micheli P., Martinez
companies. It was verified that the companies did not have V., Mason S., Marr B., Gray D., Neely A., Towards a
a common aim with this project. Although the system definition of a business performance measurement
provided several indicators, as used energy, CO2 saved, system, Int. J. Oper. Prod. Manag., 2007, 27, 784-
with real-time data collecting and dashboards, it was not 801.
possible to measure the performance, as there was no
performance target. In this case, following the framework, [2] Eccles R. G., The Performance Measurement
the first step was to set an objective, which was made Manifesto, Harv. Bus. Rev., 1991, 1, 131-137.
based on the companies’ mission/vision/values [3] Kaplan R. S., Norton D. P., The balanced scorecard--
statements, and performing documental research and measures that drive performance, Harv. Bus. Rev.,
interviews to discover what each company aimed 1999, 70, 71-9.
individually with this project. It was developed a specific [4] Bourne M., Mills J., Wilcox M., Neely A., Platts K.,
framework to assist with the selection of a set of urban Designing, implementing and updating performance
mobility indicators (Business Process Modeling (A4)) and measurement systems, Int. J. Oper. Prod. Manag.,
measures (Performance Measures (A5)). It could be used 2000, 20.
with no harm, due to the flexibility of the PMSDF. The
changes have not yet been implemented, but the use of [5] Neely A., Bourne M., Mills J., Platts K., Richards H.,
the PMSDF was helpful in the assessment and design Strategy and Performance: Getting the Measure of
phases of the project. Your Business, Vol. 2. 2002.
[6] Neely A., Richards H., Mills J., Platts K., Bourne M.,
4 CONCLUSIONS
Designing performance measures: a structured
The PMSDF was conceived to be a flexible and elegant approach, Int. J. Oper. Prod. Manag., 1997, 17,
solution, assisting not only in the design of PMS but also 1131-1152.
during the assessment and implementation stages.
[7] Franco-Santos M., Lucianetti L., Bourne M.,
Because of its flexibility, it can receive inputs from different
Contemporary performance measurement systems:
analytic methods or data sources.
A review of their consequences and a framework for
In the application cases, the framework turned out to be a research, Manag. Account. Res.,2012, 23, 79-119.
good option to work with small sized firms operations and
[8] Duarte R., Deschamps F., Loures E. R., Urban
standalone projects. Major operations of large companies
mobility indicators: a systematic literature review, in
may benefit from tailored solutions that fit better their
2016, ISERC.
needs, but even in these cases, the framework may have
results on preliminary stages or department specific [9] Slack N., Lewis M., Operations Strategy (4th Edition),
operations and projects. The framework is ready to be Pearson, 2015.
used in manufacture and service and has already been [10] Johnston R., Service operations management: from
tested in both types of operations, with no harm. the roots up, Int. J. Oper. Prod. Manag., 2005, 25,
Since the approach adopted by the framework highlights 1298-1308
mainly the strategy of the company and not the ICT
platform, it may be better accepted by businesses in less
technological markets. This hypothesis must be confirmed
by further tests.
Currently, PMSDF is being tested in a project to guide in
the assessment, design, and implementation of PMS for
multiple not-for-profit institutions. Because the framework
does not distinguish between financial and non-financial
indicators, the preliminary tests indicate that no major
changes are going to be needed.
5 ACKNOWLEDGMENTS
Authors would like to thank CNPq (National Council of
Technological and Scientific Development) for the support
in the research project through grant 307871/2012-6 and
CAPES (Coordination for the Improvement of Higher
Education Personnel) for the scholarship through grant
88887.136024/2016-00.