CMM®/CMMI® Level 4:

Quantitative Measurement
versus
Quantitative Management
Charles Weber, TeraQuest Metrics
Rick Hefner, TRW Systems

SEPG 2003 Conference

Boston, Massachusetts
February 26, 2003
Capability Maturity Model, CMM and CMMI are registered in the U.S. Patent and Trademark

Level 3 Measurement
Purpose of measurement in is to provide information
that improves decision making in time to affect the
business or mission outcome
Organization standardization drives the characteristics
of Level 3 measurement activities
• based on the standard processes, defined processes, and life
cycle models
• cover the significant attributes of all life cycle phases
• includes standard set of measures
• stored in organization’s measurement repository

Corrective action is proactive, using objective action

triggers

2
 Level 3 - Earned Value - 1
Cost Variance (CV) is the difference between actual and
budgeted costs (BCWP - ACWP)

Now
200 Cost
variance
Cost
($K) 150

100
Actual cost of
Work Performed
(ACWP)
{ CPI = .63
} (<1 over budget)

Budgeted cost of
50
work performed
(BCWP)

3 6 9 12 15 18 21
Months

Level 3 - Earned Value - 2

Schedule Variance (SV) is the difference between

budgeted cost for work performed and work scheduled
(BCWP- BCWS)
Now
200 Schedule
variance
Cost
($K) 150

100 Budgeted cost of

work scheduled
{
} SPI = .52
(<1 behind sched)

Budgeted cost of
50
work performed
(BCWP)

3 6 9 12 15 18 21
Months
4
 Level 3 - Defect Trends

Status of Severity 1 Defects

80

74
72
70
66 67 Open Last Week = 39
60 Closed This Week = 8
60
60 57 Dow ngraded = 4
57 ------
27
52 New This Week = 10
Number of Defects

50
48 Upgraded = 0
45* ------
40* 39 Open This Week = 37
40
37
38

30

18
20
18
14
14 15 18 13 12
10
10 10 11 9
10 11 10
8 10 11 8
5 9
5 6 5 8
4 4 6
4
0
4/30/98

5/8/98

5/15/98

5/20/98

5/27/98

6/3/98

6/10/98

6/17/98

6/24/98

7/1/98

7/8/98

7/15/98

7/22/98

7/29/98

8/5/98

8/12/98

8/19/98

8/26/98

9/2/98

9/9/98

9/16/98

9/2 /98
New Open (Including New) Worked off During the Week

Level 3 - Defect Densities

Defect Density
Size Defect Density
CI Defects
(KSLOC) (Defects/KSLOC)

A 44 48 1.1
B 32 60 1.9
C 36 36 1.0
D 28 33 1.2
E 34 42 1.2
F 15 46 3.1
G 9 30 3.3
Total 198 295 1.5
Project: PSM Data as of 30 June 95

6
 Gap Between Levels 3 and 4
Level In Out
4
• Process fidelity is poor
• Tailoring is too loose
• Measurement culture not mature
• Few measures tracked at event/task level
The • Missing and “dirty” data
Gap • Data inconsistent across projects
• Process performance not quantified
• Cause of performance differences unknown
• Unfocused org analysis and support
Level In Out
3

Source: Adapted from Perdue in (Paulk et al., 1995)

Characteristics of Level 4
Establish achievable quantitative project goals for
performance and product and service quality

Establish defined processes and plans that have the

capability to achieve the goals

Understand, reduce, and control process variation

Statistically predict the results of primary work efforts

(their process) on a regular basis

Statistically predict the project results against the goals

on a regular basis

Perform corrective actions so that the goals are

achieved

8
 Level 4 Measures
Problem Report Prediction Model
Problem Reports

Actual

Model
Average Prepration Time Spent
Source: Kan, 1995 Months to GA (Release)
Ave Hours per

3.00
Reviewer

2.00 Project A
1.00 UCL
0.00 LCL
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 XBAR
Code Inspection ID

Level 4 - Detection/Removal Profiles

Defect Detection Profile
180.00

160.00

140.00

120.00
Defects/KSLOC

100.00
All Projects
New Process
80.00

60.00

40.00

20.00

0.00
Req'mts Design Code Unit Test Integrate Sys Test Del 90 Days
Phase

10
Goals – Stepping Stone or Stumbling Block?
Goals need to include specification of measures and
analyses that will be used to judge whether the goals
will be / are achieved

Goals need to be expressed quantitatively or objectively

• not all goals are quantifiable

Goals need to cover a unified set of measures

Goals need to be negotiated with the stakeholders

• fact-based and data-based negotiations
• what the project will achieve

Goals represent an agreement with the stakeholders on

the measured result that will be achieved
• a commitment!

11

Setting Level 4 Goals

3
Flow-down of organizational
goals to projects is not required
Business
for Level 4, but project quality
and internal Establish
factors 5
goals are required
organization’s
goals
4
Project
requirements
Standard processes,
and objectives organization’s PCBs,
project PPBs Negotiate project
Products goals and deploy
and services 1 2 5
Project quality goals should
measure in same dimensions
7 6 as the organization’s goals,
but they do not have to satisfy
Perform Set project requirements,
process, commitments, the organizational values
project work
plans, and goals

12
 Characteristics of Level 5
Organization understands its critical business issues or
areas of concern

Organization establishes the quantitative performance

and quality improvement goals

Organization establishes the infrastructure and defines

the strategy for systematically pursuing improvements

Improvements are pursued, identified, evaluated,

piloted, and deployed to achieve the improvement goals

Three categories of process improvements

• defect and problem prevention improvements
• continuous capability improvements (individual and team)
• planned innovations
13

Level 4 versus Level 5 Goals

Level 4 and 5 projects need a stable base to succeed
• requirements, process, budget, schedule, staff, resources,
commitments, and performance and quality goals
• credible analyses that show these are consistent

Achieving organizational improvement (“stretch”) goals

is the responsibility of those who set them
• goal “flow-down” to projects can be a dangerous strategy
• the organization determines how to achieve the goals, then …
• changes to project goals are negotiated based on facts and
data

14
 Setting Level 5 Goals
3 7
6
Business 8
and internal Establish
factors Define Pilot/evaluate
organization’s
goals candidate candidate
4 changes changes
10
Project 9
requirements
and objectives
Standard processes, 11
organization’s PCBs,
project PPBs Negotiate
Products project goals
and services 1 2 11
Plan
15 5 deployment
14 13
12
Perform Set project requirements, Deploy goals and
project work process, commitments, improvements
plans, and goals

15

TRW Systems
A leading global
integrator
of complex systems
• based on information
technology and systems
engineering expertise
• integrated solutions:
architecture, Treasury Communications Intercontinental Ballistic
development System Missile program

and sustainment
Many customers and
markets in
transformation
Six Sigma – a
cornerstone of our Ohio MARCS

transformation
16
 Global Business Presence
50 States 15,000 Employees
34 Countries 2001 sales of $3.1B
Sweden
Poland
Finland

Denmark Hungary
Netherlands
United States Belgium Germany Switzerland

Austria South Korea

United Kingdom
Bosnia
France

Portugal Slovakia
Japan
Spain
Kuwait China
Italy
Taiwan
Turkey Oman
Malaysia
Brazil
Bahrain

Saudi Arabia Singapore

Israel Australia
Egypt
Guam
Diego Garcia
Marshall Islands

17

TRW Systems Business Mix

Markets Contract Type

Fixed Time & Materials/
Intelligence Civil Commercial Price Fixed Rate
17% 32% 6% 18% 16%

Defense Cost Type

45% 66%

• 2001 sales of $3.1B

• 15,000 employees
• Solutions for clients range from
architecture to development and
sustainment

18
 Six Sigma Methodologies

D- Each project must

have a business
case and sponsor
Design
Improve M- You can’t manage
New Products
and Processes Existing Processes what you don’t
that meet so that outputs meet measure
customer needs customer
requirements A- Solve the problem,
not the symptoms
I- Push for
innovations,
breakthrough
thinking
C- Who is accountable
for making the fix
stick?
Control and manage Cross-Functional Processes
to meet business goals

19

Enterprise-Wide Process Improvement

Six Sigma

ISO 9001 CMMI

Business
Business Voice
Voice of
of the
the Change
Change
Measures
Measures Customer
Customer Management
Management

Methods
Methods DMAIC
DMAIC Process
Process
and
and Tools
Tools DFSS
DFSS Management
Management

• ISO 9001 provides a quality management discipline for all project

and functional areas
• Six Sigma provides a comprehensive framework for ensuring
process improvements support corporate goals
• CMMI ensures use of industry best practices in software and
systems engineering

20
 Using Six Sigma with CMMI
For individual processes:
• CMM/CMMI identifies what activities are expected
• Six Sigma identifies how activities might be improved (more
efficient, more effective, …)
Example – Project Planning in CMMI
SG 1 Establish Estimates
SP 1.1 Estimate the Scope of the Project
SP 1.2 Establish Estimates of Project Attributes
SP 1.3 Define Project Life Cycle
SP 1.4 Determine Estimates of Effort and Cost Could fully meet CMMI
SG 2 Develop a Project Plan goals and practices,
SP 2.1 Establish the Budget and Schedule but still write poor plans
SP 2.2 Identify Project Risks
SP 2.3 Plan for Data Management
SP 2.4 Plan for Project Resources Six Sigma can be used
SP 2.5 Plan for Needed Knowledge and Skills to improve planning
SP 2.6 Plan Stakeholder Involvement
SP 2.7 Establish the Project Plan
process and write
SG 3 Obtain Commitment to the Plan better plans
SP 3.1 Review Subordinate Plans
SP 3.2 Reconcile Work and Resource Levels
SP 3.3 Obtain Plan Commitment

21

CMM/CMMI and Six Sigma Comparison

Both use same tools and methods
σ work
CMM/CMMI adds organizational focus to 6σ
σ activities increase with maturity level
Benefits of 6σ
Level Focus Constraints

5 Full toolset used to make

continuous improvements

4 Statistics used to stabilize Process must be stabilized

and predict performance before making improvements

3 Simple tools applied to Large variation in performing

standardized processes standardized processes

2 Simple tools applied to Projects use different

problems within projects processes

22
 Six Sigma and Level 4 Lessons
Typical Six Sigma Level 4 Six Sigma

Business justification for Six Achieve stable and predictable

Sigma projects must results – don’t perturb current
substantiate improvement capability

Quantified business case Level 3 capability numbers are

myth – business improvement
case fiction

Short duration projects – 4 to 6 Demonstrating stable and

months or less predictable process takes
considerable time

Full power of Six Sigma Level 3 organizations use Six

assumes a well-defined, Sigma to shore up Level 3 for
consistent process Level 4

23

Overall Lessons Learned

Level 3 metrics, measurement processes, and goal
setting are generally inadequate for Levels 4 and 5

Six Sigma is an enabler for higher maturity

• focus on data, improvement paradigm
• tying improvements to business goals
• tools and methods support the Level 4/5 analysis tasks

CMM/CMMI practices provide a framework for focusing

Six Sigma projects

Basic quality management tools (without Six Sigma

overhead) are useful and effective at lower maturity
levels

24
 Contract Information
Charlie Weber Rick Hefner
Charles.Weber@teraquest.com Rick.Hefner@trw.com
TeraQuest Metrics, Inc. TRW
P.O. Box 200490 One Space Park R2/2144
Austin, Texas, USA 78720 Redondo Beach, CA 90278
512-219-9152 (phone) 310-812-7290
512-219-0587 (fax)

site address:
12885 Research Boulevard
Austin, Texas, USA 78750

TeraQuest Web site:

http://www.teraquest.com

25