A Metric System for Measuring the Degree of Success of Managed Service Requests

Indicators are useful for measuring activity efficiency and progress toward pre-designated goals. Before implementing a KPI (Key Performance Indicator) system, an organization must have clear its mission, stakeholders and objectives. This is important because KPI are quantifiable measurements and have to be agreed beforehand in order to reflect critical success factors.

KPI can vary from organization to organization, the important thing is that they must reflect the organization's objectives and must be key to its success. Key Performance Indicators may change over time as the organization's objective change.

(For illustration purposes, a simple generic case will be taken into consideration. This case can be further analyzed for specialization purposes. In practice the following could represent a managed service request in a Call Center context.)




The objective is to control those processes that determine (from the point of view of the organization) performance efficiency and effectiveness toward a target (such as a customer). 


The steps to achieve the objective:

1) Choose which process to monitor (e.g., Process A) 

2) Identify indicators for the selected processes (e.g., Customer Satisfaction, Delivery Time) 

3) Create profile for indicators (Metrics, Family, and Source)



PROCESS -
DESCRIPTION
Family
Indicators
Metrics
Measurement criteria
Source
PROCESS A
Quality

Customer Satisfaction
Degree of success
Satisfaction level of the request/change
SLA (Service Level Agreement)
Efficiency
Delivery Time
Time
Evaluation of  the response time / delivery
Response / Specifications

Quality

Quality for this process is measured in terms of customer satisfaction:

- LREQ = represents the reception level and request comprehension (the needs of the customer)

- LRES = represents the response level to the requests

The relation LREQ / LRES defines the level of customer satisfaction in the following way:

- LRES / LREQ = 1, customer requests were satisfied (e.g., as agreed by SLA or pre-defined)

- LRES / LREQ > 1, customer requests were satisfied and expectation exceeded

- LRES / LREQ < 1, customer requests were not satisfied





Efficiency

Efficiency defines the response time from the time of receiving a request.

LEFF = (Response Time) / (Receiving a Request)




LEFF
Quality
Efficiency
= 1
Requests were satisfied as
established in the SLA.
Requests were satisfied as
established in the SLA.
< 1
Requests were not satisfied.
Requests were not satisfied.
> 1
Requests were satisfied and
expectation exceeded.
Requests were satisfied and
expectation exceeded.

Sizing Applications from User Perspective

Applications are software programs designed to perform functions for users or for other applications. Some examples of application include ATM programs for managing cash dispensing machines, word processors, database programs, web browsers, development tools, CAD/CAM programs for design work, and communication programs.

Application programs use the services of the computer's operating system and other supporting programs to perform a designated function, and programs communicate with other programs through the application program interface.

The idea here is not to illustrate how to build an application but how to make a high level estimation of its size from a user’s perspective. After having analyzed its size it’s possible to estimate the cost required to build it. 

Applications can be big and complex or small and simple. People use web applications to order products online, make reservations for hotels, apply for jobs and so on, also, use mobile applications when on the move or travelling and, furthermore, use specific applications to communicate with cash dispensing machines to take out money or for business purposes to manage stock, to meet their customer needs and other aspects of their business.

An application that needs to interface and interact with users can be characterized by a certain number of elements. These elements in a way govern the application and they are: wireframes, use cases, business rules, data tables, reports and correspondences. The size of an application can be determined by the number of these elements. For the purpose of illustrating an example, a simple hypothetical case is taken into consideration.

Application
Number of Elements
Wireframes
Use Cases
Business Rules
Data Tables
Reports
Correspondences
10
50
90
100
10
15
Estimated effort to develop one element of simple complexity level in man-days
3
5
3
3
2
2
Development Effort (man-days)
30
250
270
300
20
30
Estimated effort to capture one element during the analysis phase of simple complexity level in man-days
3
3
3
3
1
1
Analysis Effort (man-days)
30
150
270
300
10
15
Total Effort (man-days)
60
400
540
600
30
45

Legend of elements:
-Wireframes are user interfaces and it’s the way the user communicates with the application.
-Use Cases describe how a user interacts with the application in a procedural way.
-Business Rules govern the flow of procedures and data.
-Data Tables are the data required for the application to perform its function.
-Reports are printable information.
-Correspondences can be notes, messages, and emails.

N.B.- All of the above elements are managed by the application. The numbers in the table above do not refer to any industry standard, they are all hypothetical numbers just for calculation and illustration purposes.

Mathematical format:

By introducing hypothetical unit rates per day it’s possible to calculate the relative and total costs.

-Development unit rate per day = $300 
-Analysis unit rate per day = $500
Putting data in table format:

Costs
Total Effort Development in man-days
900
Developers rate per day in $
300
Developers Cost in $
270000
Total Effort Analysis in man-days
775
Analyst rate per day in $
500
Analysts Cost in $
387500
Total Cost in $
657500

All data in this example is hypothetical. For better accuracy it’s possible to add more variables and factors to this approach.

Forecasting Project Costs using Variance Analysis

One way to report on cost control and forecasting during project execution is to use the Variance Analysis method, that is, explaining the difference (or variance) between actual costs and the budgeted costs with numbers and make new estimates for completing the work. Please consult this link Earned Value Management for related literature and references.

For the purpose of making these calculations, I will use an hypothetical project example (but it could also be a task or phase).
"A company has contracted a service provider to deliver a project in 10 working days (80 hours) for the estimated cost of $10,000 and a work effort of 200 hours. The contract is Time and Material, this means that the company pays the provider for the number of hours actually required to perform the service. So, the provider has no incentive to minimize the number of hours expended on the service. The less efficient the provider is, the more money it makes!"

Summary of Time and Material Contract (representing the initial estimated baseline)

- 2 weeks duration (=10 working days)
- $10,000 cost (= budget at completion)
- 200 hours effort

Situation after 5 days

- Service Provider completed only 40% of work 
- Company received $6,500 invoice from provider (actual cost after 5 days) 
- Time sheet = 100 hours (actual number of hours spent on project after 5 days)

Using variance analysis to calculate data and to do forecasting:


After 5 days
BAC (Baseline)
Estimated after 5 days
A
(Actual)
ETC
(Estimate to Complete)
EAC
(Estimate at Completion)
% Completed
(A / EAC)
VAC
(BAC-EAC)
Work (%)
100%
50%
40%
60%
100%
40 %
0
Cost ($)
10,000
5,000
6500
6500 (b)
13000
50 %
- 3000
Duration (days)
10
5
5
7(c)
12
41.6 %
- 2
Effort (hours)
200
100
100
120 (d)
220
45.45 %
- 20


a) ETC Work is 60% because only 40% was done. The percentage change between the estimated and the actual is -20 = (4-5)/5x100, this means that the actual work is behind 20%. 

b) ETC Cost = $6,500. The %change = (5,000-6,500)/5,000 x 100 = -30%, the negative sign means that the expenditures exceeded the budgeted amount at 50% of the executed time. Hence, ETC Cost = $5,000+ 30% of $5,000= $ 6,500. 

c) ETC Duration = estimated + % of work to recuperate = 5 days + (20% of 10 days) = 7 days estimated to complete 

d) ETC Effort = estimated + work lag to recuperate = 100 + (20% of 100) = 120 hrs 

VAC (Variance At Completion) = BAC - EAC 

A graphical representation of the cost at T = 5 days:

Forecast :

- + 7 days to complete 60% of the work 
- Cost= +$ 6500 
- Duration = + 120 hours

N.B.- If the project is still not completed with the new baseline, there is a need to make new calculations for a new forecast.