Audit of

Strategic Performance
Management Information
Internal Audit
378-1-245
July 30, 2009
 FINAL Report

Table of Contents
EXECUTIVE SUMMARY _________________________________________________3
1.0 INTRODUCTION ____________________________________________________6
2.0 AUDIT OBJECTIVES AND SCOPE _____________________________________8
2.1 Audit Objectives___________________________________________________8
2.2 Audit Scope ______________________________________________________8
3.0 AUDIT APPROACH AND METHODOLOGY ______________________________9
4.0 AUDIT FINDINGS AND RECOMMENDATIONS __________________________10
4.1 Senior Management Reporting ______________________________________10
4.2 Common Reporting for Regions/Sectors_______________________________13
4.3 Integrating Performance Information __________________________________15
4.4 Information Quality _______________________________________________16
Annex A ____________________________________________________________20
Annex B ____________________________________________________________22
Annex C ____________________________________________________________23

Audit of Strategic Performance Management Information 2

 FINAL Report

EXECUTIVE SUMMARY

The Audit of Strategic Performance Management Information was identified as a priority

in the Correctional Services Canada (CSC) Internal Audit Branch 2008-09 Plan in order
to provide reasonable assurance that the management framework in place to support
performance management and corporate reporting across the Department is adequate
and effective. The focus of this audit was intended to be strategic in nature,
emphasizing CSC’s overall performance management framework to support corporate
reporting and governance.

More specifically, the objectives were to obtain reasonable assurance that key
management processes and controls are adequate and effective in the following areas:

• Senior Management Reporting - Performance management information meets the

needs of Senior Management for monitoring and strategic decision-making at the
corporate or enterprise level;
• Region/Sector Reporting - Performance management information meets the needs
of region/sector leaders for monitoring and strategic decision-making at the business
unit level;
• Integrating Performance Information - Information sources are consistent and
integrated at various levels of the organization (i.e. Senior Management, top-level
committees, Regional/Sector Management); and,
• Information Quality - Report development and quality assurance processes are in
place to ensure that quality, reliable, and standardized information are provided to
Senior Management and the Regions/Sectors.

CONCLUSION

Overall, CSC has adequate processes and internal controls for ensuring that strategic
performance information is aligned with strategic priorities and objectives and generally
contain key data elements necessary for monitoring and strategic decision- making at
the corporate level; however, the overall volume of priorities and commitments, if not
addressed, could undermine Senior Management’s ability to actively manage
performance.

• CSC has reasonable processes to ensure complete and accurate reporting against
corporate-wide strategic objectives as articulated in the Report on Plans and
Priorities, Corporate Risk Profile, HR Strategy and Transformation Plan 1 .
• Given the volume of commitments currently tracked by CSC, there would be a
benefit to the organization to rationalize and prioritize the commitments, to
streamline the metrics and reporting frequency used to monitor commitments (based

1
The transformation plan is in response to an independent review panel. See http://www.csc-
scc.gc.ca/text/organi/trnsform-eng.shtml for a copy of the “Report of the CSC Review Panel: A Roadmap
to Strengthening Public Safety” on CSC’s departmental website.

Audit of Strategic Performance Management Information 3

 FINAL Report

on priority levels), to focus on a critical subset of reports covering key aspects of

operations, human resources and finance, and once CSC’s reporting and monitoring
practices are at a sufficient level of rigour, to delegate ownership for monitoring
lower priority commitments to lower levels of the organization (i.e. lower than
EXCOM), where applicable.

While Regional Deputy Commissioners and Assistant Commissioners are

provided with regional performance information through national reports, there
are many distinct reports at the sector and regional level, with some
opportunities to streamline and standardize some reports. Further, there is a lack
of established process to cascade the corporate priorities and commitments to
the sectors and regions.

• National priorities are not systematically translated into region/sector priorities and
plans, and regions and sectors are not using a consistent approach to performance
monitoring at the business unit level. National reports provide a substantial amount
of regional data, but regions and sectors are using a wide variety of other reports as
well.

Overall, CSC has adequate processes for ensuring that national committees
interface and interact with each other to share strategic information and provide
their input on key CSC initiatives to Executive Committee (EXCOM).

• Coordination and consultation between national committees is improving but will

require continued monitoring to assess whether the new decision-making control
process at EXCOM is effective.

Senior Management currently mitigates the risk of potential inaccuracies in

performance data by drawing upon its experience and corporate knowledge to
identify anomalies in performance management data; however, there does not
appear to be an established process for capturing and documenting long-
standing corporate knowledge or building this experience/knowledge into
systems/reports.

• While the review processes seem to have been generally effective at ensuring
quality information, they rely on the experience, specialized knowledge and
corporate memory of a relatively small number of key staff.
• There are very limited documented processes, standards or protocols to assist in the
review.
• Currently, there are not complete data standards or common data definitions to
ensure that staff inputting data into CSC systems and those extracting data from the
systems have a clear understanding of how this information meets CSC’s needs for
strategic decision-making. In particular, there are concerns where a strategic
objective is measured using data that comes from unstructured fields (i.e. text
boxes) within the Offender Management System (OMS).

Audit of Strategic Performance Management Information 4

 FINAL Report

RECOMMENDATION
Recommendations have been made in the report to address these areas for
improvement. Management has reviewed and agrees with the findings contained in this
report and a Management Action Plan has been developed to address the
recommendations (see Annex C).

Audit of Strategic Performance Management Information 5

 FINAL Report

1.0 INTRODUCTION

A well functioning performance management system is critical to the success of any

organization. An integrated performance management framework combines the power
of technology, information, people and process through all stages of the management
cycle. The key components of the performance management cycle typically group into
three broad areas:

• Plan: Align the business to deliver on strategy; through planning and budgeting;
• Monitor: Run the business and monitor performance through aligned operational,
management and external reporting and analysis; and,
• Intervene: Active intervention to realign the business using management information
to manage performance and forecast future needs.

The need for this audit of the Correctional Service of Canada’s (CSC) strategic
performance information was identified as part of the Internal Audit Branch Audit Plan
for 2008-09. Specifically, the Audit Plan stated that,

“The objective of this audit will be to provide reasonable assurance that the
management framework in place to support effective performance management
and corporate reporting across the organization is adequate and effective. The
focus of this audit will be strategic in nature, emphasizing CSC’s overall
performance management framework in place to support corporate reporting and
governance.”

The audit was conducted by Internal Audit, with the assistance of Deloitte, and primarily
focused on how the management framework for strategic performance information is
carried out by and supports Senior Management acting in the following capacities:

• Commissioner: As the deputy head, sets performance objectives, approves

performance indicators and monitors actual performance;
• EXCOM members: Supports the Commissioner and serves as the top-level
management committee, reviewing performance objectives, indicators and reports,
and determining management responses;
• Performance Assurance 2 : responsible for ensuring mechanisms are in place to
measure, quantify, monitor and analyze CSC's performance;
• National Committee Chairs: responsible for the management of Committees such as
Transformation, National Health Services, and Policy and Communications,
including the integration of strategic information from across CSC in support of
decision-making and performance management;

2
The Performance Assurance Sector, which included the Performance Management Branch, was
merged with the Policy and Research Sector in 2008-09. The new sector was renamed as the Policy
Sector and the Assistant Commissioner Policy now has responsibility for performance management.

Audit of Strategic Performance Management Information 6

 FINAL Report

• Regional Deputy Commissioners: responsible for CSC operations, managing

performance against CSC and regional objectives within their respective regions;
and,
• Assistant Commissioners: responsible for establishing indicators for performance,
systems and standards for the collection of performance information and for
monitoring and supporting RDC’s in managing performance within their respective
policy areas.

CSC’s results and performance objectives are supported by a range of controls that
permit management to establish and review performance objectives and corresponding
performance targets and indicators. Controls in this area also encompass the
processes to monitor financial and operational performance on an ongoing basis and
the degree to which performance results are fed back into the planning process and
articulated in key corporate reports. The performance management system should be
used as a vehicle through which changes or risks to the internal and external operating
environment are proactively reviewed and considered. Performance management is a
key management tool, supporting all of CSC’s strategic outcomes and priorities; more
directly, it supports the stated priority of “strengthening management practices”, as
articulated in CSC’s Report on Plans and Priorities. Effective performance management
and reporting are key to improving management practices, good governance,
transparency and accountability, and serve as control functions for important corporate
risks (e.g. security, human resources management, infrastructure management and
planning and information management/technology investment).

On an annual basis, CSC is assessed under the “Management Accountability

Framework (MAF)” which sets out Treasury Board's expectations of senior public
service managers for good public service management. The MAF is structured around
10 key elements that collectively define "management" and establish the expectations
for good management of a department or agency. The following table provides a
summary of MAF assessment ratings over the last 3 years in areas relating to
performance information and related management practices which provides context for
the audit in the CSC environment:

HIGHLIGHTS FROM CSC’S MAF

2006 RATING 2007 RATING 2008 RATING
ASSESSMENT
Utility of the Corporate Performance
Framework
(e.g. PAA alignment with CSC’s mandate, Acceptable Acceptable Acceptable
measurability of strategic outcomes)
Effectiveness of the Corporate
Management Structure
(e.g. alignment and integration of corporate Acceptable Strong Strong
business plans with corporate priorities)
Quality of Performance Reporting
(e.g. DPR, RPP including source of data, Acceptable Acceptable Acceptable
quality of information, and strategic context)
Effectiveness of Corporate Risk
Management Acceptable Acceptable Acceptable
(e.g. corporate risk profile)

Audit of Strategic Performance Management Information 7

 FINAL Report

2.0 AUDIT OBJECTIVES AND SCOPE

2.1 Audit Objectives

The objective of the audit was to obtain reasonable assurance that key management
processes and controls are adequate and effective in the following areas:

a) Senior Management Reporting - Performance management information meets the

needs of Senior Management for monitoring and strategic decision-making at the
corporate or enterprise level;
b) Region/Sector Reporting - Performance management information meets the needs
of region/sector leaders for monitoring and strategic decision-making at the
business unit level;
c) Integrating Performance Information - Information sources are consistent and
integrated at various levels of the organization (i.e. Senior Management, top-level
committees, Regional/Sector Management); and,
d) Information Quality - Report development and quality assurance processes are in
place to ensure that quality, reliable, and standardized information are provided to
Senior Management and the Regions/Sectors.

Specific criteria related to each of the objectives are included in Annex A.

2.2 Audit Scope

As part of this engagement, Internal Audit performed a high level risk assessment of
CSC’s strategic performance information through a preliminary survey. Based on the
results of the preliminary risk assessment, an audit program was designed to obtain
reasonable assurance that the management framework in place to support effective
performance information across the Department is adequate and effective. Since the
audit had a strategic focus, not all areas of the Performance Management Framework
were covered as the focus was on key reports and processes. In addition, because
financial reporting was the subject of a separate audit (the Audit of Financial Planning,
Budgeting and Monitoring” 3 ), it was not an area of focus in this audit.

During the preliminary risk assessment, a number of aspects of performance

management information were assessed through interviews and document review. The
findings were grouped into seven types of risk. Using risk rating criteria, the focus for
the audit was reduced to the following interrelated areas: Senior Management
Reporting; Integrating Performance Information; and Information Quality. In addition,
the audit was focused primarily on the following sources of strategic performance
information:

3
See http:// www.csc-scc.gc.ca/text/pa/iapp-toc-eng.shtml for a copy of the “Audit of Financial Planning,
Budgeting and Monitoring” on CSC’s departmental website.

Audit of Strategic Performance Management Information 8

 FINAL Report

• Report on Plans and Priorities (RPP) - outlines CSC priorities and commitments and
maps these to the CSC Program Activity Architecture (PAA);
• Mid-year and year-end performance reviews - covers all 132 RPP commitments and
describes whether commitments are on time and on budget, summarized on a 4
quadrant grid. The reviews identify each commitment and milestone, status,
explanation of variances, and Office of Primary Interest (OPI) responsible for the
priority;
• Corporate Risk Profile (CRP) – depicts the corporate risks identified for CSC in a risk
matrix. Identified risks appear in descending order of residual (total) risk exposure
with a weighted value;
• Human Resources (HR) Strategic Plan – relates to the three-year Strategic Plan for
Human Resource Management. The plan focuses on four priorities for CSC:
strengthening its human resource management practices; building an effective
representative workforce; providing learning, training and development; and
improving workplace health and labour relations. The plan also includes
measurement strategies for HR priorities; and
• Transformation Plan – describes progress on the Transformation Initiative,
established to implement the CSC Review Panel's 109 recommendations.

The Atlantic and Ontario regions as well as the Correctional Operations and Programs
(COP) and HR sectors were selected for review in order to provide a reasonable sample
of business units.

The audit scope did not include substantive testing of data/information contained within
various strategic performance reports, but instead focused on processes and controls
designed to ensure the quality of data.

3.0 AUDIT APPROACH AND METHODOLOGY

The approach and methodology used was consistent with the Internal Audit standards
as outlined by the Institute of Internal Auditors, and was aligned with the Internal Audit
Policy for the Government of Canada.

A planning phase was completed to identify key risks related to CSC`s strategic
performance information which should be explored further in the conduct phase.

In the conduct phase, evidence was collected using the following techniques:

• Interviews: A total of 17 interviews were conducted at National Headquarters

(NHQ) and regional levels – 6 Regional Deputy Commissioners/Assistant
Commissioners, 2 Regional Performance Analysts, 3 Sector Performance Analysts
and 2 representatives of the Performance Assurance Branch and 4 other interviews
with various CSC staff members.
• Review of Documentation: Relevant documentation such as process
documentation, procedures and sample performance reports at the National, Region
and Sector levels.

Audit of Strategic Performance Management Information 9

 FINAL Report

• Walkthroughs: Step by step walkthroughs of key reporting and quality control

processes to document steps, identify roles and responsibilities and document
evidence of compliance with key controls.

Upon completion of the conduct phase, the reporting phase entailed analysis of the
results of the audit procedures conducted, synthesis of results, development of a draft
report, review and validation with management, and the finalization of the report. A table
summarizing sites examined is included in Annex B.

4.0 AUDIT FINDINGS AND RECOMMENDATIONS

4.1 Senior Management Reporting

During the planning phase, concerns were raised that there did not appear to be an
established, scheduled and focused reporting suite for Senior Management or the top-
level committees used for monitoring the Department’s strategic initiatives. The lack of a
shared reporting approach and standardized, focused reporting suite would increase the
risk that executive and committee decisions are not being made on consistent data, that
points of integration are not given due consideration, and that the most critical priorities
are not given sufficient attention.

We assessed the degree to which performance management information is meeting the

needs of Senior Management for monitoring and strategic decision-making at the
enterprise level. To facilitate this assessment, we have assumed, for the purposes of
this audit, that the strategic commitments of CSC were accurately captured in existing
planning documents. Based on the findings in the Preliminary Risk Assessment, it was
decided to focus on the Report on Plans and Priorities (RPP), Corporate Risk Profile
(CRP), HR Strategy and Transformation Plan.

We expected to find that CSC has processes and controls at the enterprise level to
ensure that Senior Management is provided with performance information to monitor
progress against stated strategic priorities and objectives as well as to assist in strategic
decision-making.

Overall, CSC has adequate processes and internal controls for ensuring that
strategic performance information is aligned with strategic priorities and
objectives and generally contain key data elements necessary for monitoring and
strategic decision-making at the enterprise level; however, the overall volume of
priorities and commitments, if not addressed, could undermine Senior
Management’s ability to actively manage performance.

Audit of Strategic Performance Management Information 10

 FINAL Report

Based on our interviews and documentation review, Executive Committee (EXCOM)

members receive and use a core set of strategic performance management information
including:

• Departmental Performance Report (DPR);

• Mid-year and year-end performance reviews;
• Corporate Risk Profile (CRP) reviews;
• HR Strategic Plan Updates/Progress Reports; and
• Transformation Plan and Updates/Progress Reports.

The audit observed good manual processes in the Office of the Commissioner and the
Performance Assurance group to track priorities and commitments related to the RPP
and CRP in particular, and to proactively manage the production of performance
information for discussion at EXCOM at appropriate intervals. Forward agenda
management appears to be effective at ensuring follow-up on commitments. In
addition, a process outlined in the “Process for Decision-Making” deck has been
implemented that requires EXCOM members to complete a decision-making checklist
before submissions are brought before EXCOM. The sponsoring EXCOM member is
required to address all significant issues in the checklist including an explanation of how
the submission relates to CSC priorities, whether consultations have been made and
how it links to other CSC initiatives.

Performance Assurance provides effective leadership to CSC in tracking and reporting

on RPP commitments and corporate risks. Reasonable processes were reported by
leaders in HR and Transformation on the preparation of progress reports against their
strategic plans.

We noted, however, that the overall number of items being tracked is substantial. For
example, there are 132 RPP commitments, 16 corporate risks with mitigation strategies,
109 recommendations from the Panel Report relating to the Transformation agenda,
Strategic HR Plans, an Aboriginal Accountability Framework and other plans, each with
numerous action and results indicators. CSC also tracks other operational
commitments including those to external stakeholders such as the Office of the
Correctional Investigator (OCI). As such, there are hundreds of commitments tracked
on an annual basis at CSC by National Head Quarters, sectors and regions. This
multiplication of commitments is mitigated somewhat by the fact that CSC leaders
consistently cite the RPP as the top-level statement of priorities for the Service and that
RPP commitments have been grouped under five higher-level Commissioner’s
priorities. However, it remains a challenging volume of items to be tracked at the
strategic level.

An emerging practice at the region/sector level which may provide a useful model for
CSC overall was noted in COPS where the Assistant Commissioner has begun
planning an initiative to determine which reports and monitoring activities no longer
provide a benefit to CSC and should be discontinued. The expected outcome of this

Audit of Strategic Performance Management Information 11

 FINAL Report

exercise is to enable COPS to focus its limited resources more effectively on areas
relating to the Sector’s key priorities.

The audit team also noted that the performance indicators tracked include a significant
number of activity (e.g. assessments completed) and project-type metrics (e.g. budget,
timelines, milestones) and a more limited number of outcome indicators. For example,
the Transformation Team currently monitors progress by tracking the number of work
plans that have been completed by CSC staff; these metrics are important, however,
they will not enable CSC to assess achievement of the ultimate outcomes of these
initiatives. CSC needs to clearly establish the long-term basis of measurement to
monitor the results of its initiatives and investments (specifically for high
importance/impact objectives), in addition to resources, activities and project
milestones.

The EXCOM ‘Dashboard’ is considered a key ongoing tool for CSC to monitor the
status of RPP commitments (on time and on budget), supplementing the periodic hard-
copy reports provided to EXCOM members. The Dashboard has direct links to more
detailed performance information on specific items, the description of the status and any
remedial plans. It is updated on a quarterly basis by the Performance Assurance group.
Yet, for the EXCOM members interviewed, the Dashboard is not widely used as it is
perceived as too high level in the way it presents RPP commitments (on time and on
budget) and National, HR and RPP results (critical, caution, normal). In addition, some
individuals stated that they are not familiar with how to effectively use it. Improvements
to the EXCOM Dashboard and/or education on its use are required to make it more
relevant, useful, accessible and sufficiently results-oriented for Senior Management.

CONCLUSION

Overall, CSC has a strong discipline in reporting against commitments at the corporate
level but CSC is facing a data volume challenge and there are opportunities to improve
some of the department’s key reporting tools. Performance Assurance 4 has a number
of initiatives underway to improve performance monitoring at CSC. Notably:

• Performance Assurance has updated the Performance Management Framework to

consolidate RPP commitments, CRP issues and other commitments into one
document under the Program Activity Architecture, which proposes to use indices to
monitor performance where possible. The effective use of indices will help provide
to Senior Managers a richer, more representative picture of performance and
outcomes; and
• Performance Assurance is moving to expand the CRP to include functional risks
from sectors and regions. This will eliminate the need for separate tracking of risk
profiles at different levels of the organization and help in integrating goal setting at
the national, regional and sector levels.

4
The Performance Assurance Sector, which included the Performance Management Branch, was
merged with the Policy and Research Sector in 2008-09. The new sector was renamed as the Policy
Sector and the Assistant Commissioner Policy now has responsibility for performance management.

Audit of Strategic Performance Management Information 12

 FINAL Report

RECOMMENDATION 1
The Assistant Commissioner, Policy in collaboration with the Assistant Commissioner,
Human Resource Management, the Assistant Commissioner, Correctional Operations
and Programs, Assistant Commissioner, Corporate Services and Director General,
Executive Services should:

• undertake an initiative to rationalize and prioritize the commitments made by CSC

with a view to streamline the metrics and reporting frequency used to monitor
commitments (based on priority levels) and to focus on a critical subset of reports
covering key aspects of operations, human resources and finance. Once CSC’s
reporting and monitoring practices are at a sufficient level of rigour and discipline,
consideration should also be given to delegating ownership for monitoring lower
priority commitments to lower levels of the organization (i.e. lower than EXCOM),
where applicable.

RECOMMENDATION 2
The Assistant Commissioner, Policy should solicit user feedback on the EXCOM
dashboard tool with a view to make it more pertinent to their needs.

4.2 Common Reporting for Regions/Sectors

As with Senior Management Reporting, concerns were expressed that different
region/sector senior managers are not basing their strategic decisions on a core
platform of information and that there is a lack of sharing of information between
regions, sectors and functions. In a department where the vast majority of people and
budget are in the regions, it is especially critical that corporate objectives are clearly
assigned to the regions, and that the regional leaders monitor their performance against
these objectives.

We expected to find that CSC Regional Deputy Commissioners (RDCs) and Assistant
Commissioners (ACs) receive and use the same core performance management
information, adjusted for regional/sector differences, to manage strategic performance
at the business unit level.

While RDCs and ACs are provided with regional performance information through
national reports, there are many distinct reports at the sector and regional level,
with some opportunities to streamline and standardize some reports.

A variety of reports provide performance information by region or sector including: mid-

year and year-end reviews; performance agreements; CRP mid-point and year-end
reports; and HR Strategy Reports. Our review of region and sector performance
reports, however, found a wide variety of other report formats, performance indicators

Audit of Strategic Performance Management Information 13

 FINAL Report

and areas of emphasis. There is an opportunity to streamline and standardize some

reports (e.g. trend analysis reports) across regions/sectors, allowing region and sector
performance analysts to focus on truly unique reporting and analysis needs.

The lack of a standardized set of regional reports creates a risk that region/sector senior
managers may not be basing their strategic decisions on a common platform of
information (e.g. data definitions, time frames, etc.). In addition, there is a risk of
duplication of effort as similar reports are being produced in different regions/sectors.
Examples of region and sector reports that may benefit from greater standardization
and consistency in content include mid-year performance reviews, 10 year trends for
releases, 10 year performance summary reports, population management reports, and
reports on year-to-date output indicators. It is expected that the implementation of
recommendations # 1, 2 and 4 will address these issues.

There is a lack of established process to disseminate corporate strategic

priorities (i.e. from the RPP) and performance indicators in a way that is relevant
to regional operational managers (e.g. Wardens). While the performance
agreement process is used as a proxy in some areas, it is not considered an
effective tool for monitoring regional and national priorities in the
regions/sectors.

Part of the explanation for the divergence in approaches to performance reporting may
be that there does not appear to be consistent planning processes at the region/sector
level where national priorities can be translated into specific sub-objectives for each
region/sector. Additionally, corporate objectives are not systematically allocated to the
regions/sectors – for instance, a national target to reduce recidivism in the aboriginal
population is not formally sub-divided or prioritized amongst the regions based on
criteria such as aboriginal population characteristics, aboriginal programs, etc. As a
result, although there was strong knowledge of the national priorities, regional personnel
could not consistently say what their regional target or goal was to support a given
national objective.

A good practice in this regard is the Prairie Region “National Plan Matrix” for HR, which
expressly documents regional HR targets and commitments aligned with the national
HR Strategy deliverables and measurement strategies. Some of these regional
commitments are then added to individual performance agreements.

A related issue concerns the role of performance analysts in the regions. While only
Ontario and Atlantic were examined in detail, performance analysts appear to play
different roles in different regions. However, in a recent decision (April 2009) by
EXCOM, this issue was addressed. EXCOM approved the amalgamation of Policy and
Planning with Performance Assurance at the regional level with Regional Administration
Policy and Planning as lead. The model establishes a clear separation between the
operations and the performance measurement functions and will provide for consistency
across regions.

Audit of Strategic Performance Management Information 14

 FINAL Report

CONCLUSION

While national reports provide performance information for the regions/sectors, more
could be done to support their strategic performance information needs, notably by
building more effective processes to cascade corporate commitments and objectives to
regions and sectors, and ensuring there is appropriate support within the region to
monitor performance. At present, the regions/ sectors lack a comprehensive core
platform of regional/sector performance information that integrates strategic,
operational, Finance and HR perspectives.

RECOMMENDATION 3
The Assistant Commissioner, Policy should lead the development of a formal process to
cascade national strategies, priorities and commitments to the sectors and regions in
order to provide a stronger base for performance management at the region/sector
level.

4.3 Integrating Performance Information

At the strategic level, CSC relies on a set of executive committees with different
functional responsibilities (Operations, Finance, HR, Health Services, Policy and
Communications, Transformation, the Audit Committee, and others) to manage issues,
analyze options and make recommendations to EXCOM and the Commissioner. The
preliminary risk assessment raised concerns that “silo’ed” analysis and performance
information was affecting the quality of decision-making. We reviewed the way different
decision makers and committees at the national level obtain and use information as
they consider strategic decisions such as planning and policy making. In particular, we
focused on the National HR Committee and the Transformation Committee.

We expected to find that established processes and internal controls exist to ensure that
national committees interface and interact with each other to share strategic
information, consult on key decisions, and solicit input on key CSC initiatives.

Overall, CSC has adequate processes and internal controls for ensuring that national
committees interface and interact with each other to share strategic information and
provide their input on key CSC initiatives at EXCOM.

During the preliminary risk assessment phase of this audit, concerns were noted about
past proposals raised by national committees that did not incorporate important regional
or sectoral considerations and information. However, the audit identified several factors
which seem to mitigate this risk to an acceptable level. First, this risk is mitigated by the
fact that national committees do not have decision-making power – that is reserved for
EXCOM or the Commissioner. Second, the Director General, Executive Services has
recently implemented a decision-making checklist, as described in the “Process for
Decision Making – EXCOM” deck, that requests verification that appropriate

Audit of Strategic Performance Management Information 15

 FINAL Report

consultations have taken place prior to an issue being presented to EXCOM. Third, the
broad membership of EXCOM, with representation from all regions and all functional
areas ensures that multiple perspectives are considered and that concerns with any
proposals or the supporting performance information presented by national committees
can be raised. This integration has been further strengthened by the amalgamation of
the National HR and National Finance committees into EXCOM during the fall of 2008.

At this point, the processes behind the new decision-making checklist are informal,
making it difficult to assess the effectiveness with which performance management
information has been shared between national committees. It would be advisable to
assess the impact of the decision-making process within the next six to twelve months
to see if more formal supporting processes are required, such as rules around the
timing of inter-committee consultations.

CONCLUSION

Given that decisions are made by EXCOM on the recommendation of a national

committee, and given the implementation of the new decision-making process (to
ensure proposals and performance information presented to EXCOM by national
committees have been consulted on and use integrated information), overall the
process to ensure national committees interface appear to be adequate. Nonetheless,
at an appropriate time, the results of the new decision-making process should be
assessed to determine its effectiveness since implementation in late 2008. In addition,
for national committees outside of EXCOM (e.g. National Health Services, Policy and
Communications, Transformation, Audit Committee and others), there is some evidence
that national committees interface and interact with each other to share strategic
information; however, given the informality of this process, the degree to which
integration takes place effectively between these committees is unclear and it could be
better captured by the implementation of the decision-making process.

RECOMMENDATION 4
The Director General, Executive Services should conduct an assessment of the impact
of the new decision-making process within the next six to twelve months to confirm if it
is effective in ensuring appropriate consultations are being completed and integrated
information is being used.

4.4 Information Quality

With its highly operational mandate, CSC maintains extensive systems and processes
to track a wide variety of information and performance metrics. While this provides CSC
with a rich source of data from which to draw strategic performance information,
concerns were raised with respect to the quality and consistency of data used to enable
management decision-making, with specific references being made to OMS data
quality. Contributing factors noted in the Preliminary Risk Assessment included a lack of

Audit of Strategic Performance Management Information 16

 FINAL Report

clear understanding of management’s information needs as well a lack of depth in the

use of data sources, sets and assumptions used in the report preparation process. To
understand potential areas of weakness in this area, the audit team reviewed key
controls and activities relating to processes for extracting data, determining how
management’s data needs are addressed, and the quality assurance process followed
to ensure that data is relevant, accurate, contextualized and integrates all information
relevant to management. Our focus was to examine quality assurance processes rather
than system automated controls of CSC information systems.

We expected to find quality assurance processes and internal controls to ensure that
strategic performance information includes key assumptions, data sources, contextual
analysis and integrates cross-functional information before being released to CSC
executives for strategic decision-making and monitoring purposes, and that
assumptions, sources and cross-functional issues are clearly disclosed in the reports.

Senior Management currently mitigates the risk of potential inaccuracies in

performance data by drawing upon its experience and corporate knowledge to
identify anomalies in performance management data; however, there does not
appear to be an established process for capturing and documenting long-
standing corporate knowledge or building this experience/knowledge into
systems/reports.

Performance information generated in the regions, sectors and by Performance

Assurance undergo multiple reviews before being passed on to EXCOM, the
Commissioner or externally. The more public the information is expected to be, the
higher the level of scrutiny applied. The review process is not standardized across
regions and sectors, but seems to minimally include review by one to two performance
analysts and an AC/RDC. While these review processes seem to have been generally
effective at ensuring quality information, they rely on the experience, specialized
knowledge and corporate memory of the performance analysts and senior managers.
There are limited system level controls and very limited documented processes,
standards or protocols to assist in the review process. As such, these management
controls would be at significant risk with the turnover of a relatively small number of key
staff. And even with these management controls, there remains a risk that information
quality issues will still be faced periodically unless more rigour and formalized
processes are put in place.

Currently, there are not complete data standards or common data definitions to ensure
that staff inputting data into CSC systems and those extracting data from the systems
have a clear understanding of how this information meets CSC’s needs for strategic
decision-making. In particular, there are concerns where a strategic objective is
measured using data that comes from unstructured fields (i.e. text boxes) within the
Offender Management System (OMS). It must be acknowledged that OMS was not
originally designed to generate statistics but rather to automate the work of CSC staff.
The system was largely designed by end-user input; however, there has been
significant change and redesign of how staff conduct their work over the years which

Audit of Strategic Performance Management Information 17

 FINAL Report

has contributed significantly to the challenges of extracting data. In addition, OMS does
not have the capacity to generate all the data needed for statistical reporting and work
around methods have been and will remain necessary given the design of OMS. All of
these factors have contributed to the lack of complete data standards and data
definitions in OMS.

Significant risks to base data quality were noted consistently during interviews with CSC
staff. This was true of both HR data and operational performance data drawn from
OMS. A number of factors were cited as contributing to this risk:

• Lack of training for front-line users who are responsible for data input;
• Lack of common or enforced data definitions;
• Lack of documented processes/analysis to provide a verification trail; and
• Limited resources in performance analysis/data quality.

The small community of performance analysts at CSC provides a very valuable function
to the department. By combining their knowledge of correctional operations and their
understanding of OMS and other data bases, they are able to mitigate many of the risks
noted above. However, this group is small (six to twelve staff) and made up primarily of
employees approaching retirement. Succession planning for this group will become
critical in the next five years.

In addition, for the last 2 years, CSC’s Change Advisory Board (CAB) has served to
assess the impact of OMS changes on related systems, data queries and reports, and
to ensure that affected parties are informed about these changes. CAB includes
representation from the Performance Assurance and OMS group. Although this
established process is in place, there appears to be opportunity for improvement for
coordination between the OMS systems team, the Performance Assurance Group, and
Regional/Sector Performance Analysts as expressed in multiple interviews.
Performance Assurance analysts expressed concern that changes were sometimes
being made to the data structure in OMS without their knowledge or input. All changes
to OMS source data create additional effort to update queries and reports, but changes
that are not communicated create a risk that unreliable data could be unknowingly
provided to decision-makers or external stakeholders. Given the impact that OMS
changes have on CSC, it is important that the change management process
administered through CAB is followed to minimize operational inefficiencies and risks to
reporting integrity.

It should also be noted that Performance Assurance has established a small data
quality team. This team will manage data quality more proactively, but the level of
resources is limited. They run queries to test data in known areas of weakness and are
beginning to write some procedures to improve data entry and quality testing.

Audit of Strategic Performance Management Information 18

 FINAL Report

In the area of HR information, HR has resolved a number of data definition questions by

adopting standard definitions used in the broader public service. This will allow them to
focus more effort on other aspects of information quality and interpretation 5 .

CONCLUSION

Although quality assurance processes and internal controls exist to ensure that strategic
performance information includes key assumptions, data sources and analysis,
systemic risks are evident within the key data holdings of the department. A number of
procedures are in place to mitigate the risks related to information quality, however,
these mitigating strategies will be more sustainable if they are documented and there is
a commitment to maintain a core group of individuals with the specialized knowledge
and skills related to operational performance information.

RECOMMENDATION 5
The Assistant Commissioner, Policy should document and implement a comprehensive
approach to quality assurance (QA) on strategic operational information, including the
following:

• Documentation standards – analytical techniques, data sources, assumptions,

cross-functional issues, etc.;
• Roles and responsibilities – clarify roles and responsibilities for data quality;
• Training requirements – improve training for data input and analysis for front-line
staff; and
• Regional processes – standards to ensure consistency in report development and
QA processes in the regions.

5
It should be noted that the Internal Audit Branch plans to conduct an Audit of HR Data Integrity in
2009-10.

Audit of Strategic Performance Management Information 19

 FINAL Report

Annex A

AUDIT OBJECTIVES AND CRITERIA

Based on the Preliminary Risk Assessment, it was determined that the audit should
focus on Senior Management Reporting as well as criteria related to Integrating
Performance Information, and Information Quality. The objective of the audit is to obtain
reasonable assurance that key management processes and controls are adequate and
effective in the following areas:

a) Performance management information meets the needs of Senior Management and

the Regions/ Sectors for strategic decision-making;
b) Information sources are consistent and integrated at various levels of the
organization (i.e. Senior Management, top-level committees, Regional/Sector
Management); and,
c) Report development and quality assurance processes are in place to ensure that
quality, reliable, and standardized information are provided to Senior Management
and the Regions/Sectors.

The following table maps the control objectives mentioned above to audit criteria and
related key audit procedures for gathering evidence.

OBJECTIVES CRITERIA
1. Performance management information meets the 1.1 Established processes and internal controls
needs of Senior Management and the Regions/ exist to ensure that key performance information
Sectors for strategic decision-making. provided to Senior Management is aligned with
stated strategic priorities and objectives and
include elements to assist in monitoring and
strategic decision-making.

1.2 CSC Regional Deputy Commissioners (RDCs)/

Assistant Commissioners (ACs) receive and use
the same core performance management
information, adjusted for regional/sector
differences.

2. Information sources are consistent and 2.1 Established processes and internal controls
integrated at various levels of the organization (i.e. exist to ensure that National Committees interface
Senior Management, top-level committees, and interact with each other and share strategic
Regional/Sector Management) information and their input on key CSC initiatives.

Audit of Strategic Performance Management Information 20

 FINAL Report

3. Report development and quality assurance 3.1 Quality assurance processes and internal
processes are in place to ensure that quality, controls exist to ensure that strategic performance
reliable, and standardized information are provided information includes key assumptions, data
to Senior Management and the Regions/Sectors. sources, and analysis and integrates cross-
functional information before being released to
CSC executives for strategic decision-making and
monitoring purposes. Assumptions, sources and
cross-functional issues are clearly disclosed in the
reports.

Audit of Strategic Performance Management Information 21

 FINAL Report

Annex B 

LOCATION OF SITE EXAMINATIONS

REGION SITES
• Performance Management
• Correctional Operations and Programs
NHQ Sector
• Human Resource Management
• Regional Headquarters
Atlantic • Performance Management
• Regional Headquarters
Ontario • Performance Management

Audit of Strategic Performance Management Information 22

 FINAL Report

Annex C

AUDIT OF STRATEGIC PERFORMANCE MANAGEMENT INFORMATION

MANAGEMENT ACTION PLAN

PLANNED
RECOMMENDATION ACTION SUMMARY OPI COMPLETION
DATE
Recommendation #1:
The Assistant Commissioner, Policy in 1. Creation of a working group under DGPPP 2009-08-15
collaboration with the Assistant Commissioner, DGPPP
Human Resource Management, the Assistant
Commissioner, Correctional Operations and
Programs, Assistant Commissioner, Corporate
Services and Director General, Executive
Services should:

• undertake an initiative to rationalize and 2. Creation of a total list of commitments Working 2009-10-01
prioritize the commitments made by CSC Group
with a view to streamline the metrics and
reporting frequency used to monitor
commitments (based on priority levels) and
to focus on a critical subset of reports
covering key aspects of operations, human
resources and finance.
3. Agreement on a system of prioritization, Working 2010-01-31
by categories, and application of the Group
system to the list

4. Agreement on the relevant metrics for Working 2010-01-31

each category or item on the list Group

5. Agreement on the frequency of review Working 2010-01-31

for each category and on a schedule for Group
phased release of products.

6. Development of a list of operations, Working 2010-01-31

human resource and finance reports Group
based on the list of commitments.

7. Obtain Excom approval for results 1 ACP 2010-02-28

through 6.

8. Implement process and reporting EXCOM 2010-04-01

9. Monitor process through Excom agenda (report) 2010 -09

and records of decision. Report on ACP 2011-09
progress and review outcome, annually, (review) 2012-09 etc.
at the Strategic Planning session. EXCOM

Audit of Strategic Performance Management Information 23

 FINAL Report

PLANNED
RECOMMENDATION ACTION SUMMARY OPI COMPLETION
DATE
• Once CSC’s reporting and monitoring 1. Completion of first review of process EXCOM 2010-09
practices are at a sufficient level of rigour and results
and discipline, consideration should also
be given to delegating ownership for
monitoring lower priority commitments to
lower levels of the organization (i.e. lower
than EXCOM), where applicable.
2. Analysis of the findings with a view to Working 2010-10
recommending options for a matrix of Group
accountability for monitoring and
reporting.

3. Consultation with Sectors/ RHQ on ACP 2010-12

options

4. Excom approval of an option for a ACP 2011-02

matrix.

5. Implement matrix and monitoring. ACP 2011-04-01

6. Annual review of the matrix at Strategic (report) 2011-09

Planning session. ACP 2012-09 etc.
(review)
EXCOM
Recommendation #2:
The Assistant Commissioner, Policy should 1. Consultations are held with Sector, DGPPP
solicit user feedback on the EXCOM RHQ, and Operations managers to Ongoing
dashboard tool with a view to make it more refine their management reporting 2009-2011
pertinent to their needs. requirements

2. Existing reporting tools are reviewed for DGPPP 2009- 12-31

opportunities for short-term
improvements

3. Develop schedule of phased release of DGPPP 2009-12-31

improvements to systems. Release
improvements according to schedule.

4. Consultations with IM/IT Units for DGPPP 2009-12-31

technological options for long-term
improvements

5. Develop options, including costing DGPPP 2010-03-31

estimates for review by IMTSC

6. Collaboratively present options to ACP/ 2010-06-15

Excom for approval and possible ACCS
funding

7. Implement approved long-term option DGPPP 2011-03-31

Audit of Strategic Performance Management Information 24

 FINAL Report

PLANNED
RECOMMENDATION ACTION SUMMARY OPI COMPLETION
DATE
8. Monitor usage and client satisfaction. DGPPP Annually or
Modify as needed. following major
policy
/program
change
Recommendation #3:
Assistant Commissioner, Policy should lead the 1. Develop a list of operations, human Working 2010-01-31
development of a formal process to cascade resource and finance reports based on Group
national strategies, priorities and commitments the list of commitments.
to the sectors and regions in order to provide a
stronger base for performance management at
the region/sector level.
2. Develop process with Regional ACP 2010-01-31
Administrators Policy and Planning for
cascading products downwards and
reporting upwards.

3. Obtain Excom approval of an option for ACP 2011-02

a matrix of monitoring accountability
including some assignments to lower
levels in the organization.

4. Deliver training and support to sectors DGPPP 2010-03-31

/regions in using existing performance ongoing with
reporting tools. new staff

5. Implement approved long-term option DGPPP 2011-03-31

for delivery of performance data.
(Recommendation 2 Step 7)

6. Gradually integrate processes into DGPPP FY 2010-12

ongoing Business Planning and and ongoing
reporting processes such as RPP, DPR, according to
MAF Assessment, etc. with full product
integration in 2011-12 documents. schedule for
RPP, DPR,
MAF etc.
7. Monitor usage and client satisfaction. DGPPP Annually or
Modify as needed. following major
policy
/program
change
Recommendation #4:
The Director General, Executive Services 1. Discuss the Decision-Making Process A/DGES 2009-10-30
should conduct an assessment of the impact of at an EXCOM meeting to seek
the new decision-making process within the feedback on efficiency and
next six to twelve months to confirm if it is effectiveness.
effective in ensuring appropriate consultations
are being completed and integrated information
is being used.

Audit of Strategic Performance Management Information 25

 FINAL Report

PLANNED
RECOMMENDATION ACTION SUMMARY OPI COMPLETION
DATE
2. Complete an analysis of a sample of A/DGES 2009-11-30
EXCOM submissions for decision-
making since the promulgation of the
Decision-Making Process.

3. Consult on the results of draft analysis. A/DGES 2010-01-30

4. Present the analysis to EXCOM for A/DGES 2010-03-30

discussion/decision.

5. Implement, and post on Infonet, the A/DGES 2010-04-30

EXCOM decision to maintain or adjust
the Decision-Making Process.

Recommendation 5:
The Assistant Commissioner, Policy should 1. Collaboration with Sectors, Regions and DGPPP/ 2010-03-31
document and implement a comprehensive IM/IT to develop role and responsibility DGIMIT Ongoing
approach to quality assurance (QA) on standards for data definition, data
strategic operational information, including the quality, extraction and analysis, and
following: correcting faulty data.

• Documentation standards – analytical 2. Develop a prioritized list of data quality DGPPP/D 2010-03-31
techniques, data sources, assumptions, issues and a phased schedule for GIMIT/
cross-functional issues, etc.; implementing data quality reports and Sectors/
corrections. Regions

• Roles and responsibilities – clarify roles 3. Include data quality roles and DGPPP 2011-12-31
and responsibilities for data quality; accountability in policy and procedure Ongoing
materials as Policy Task Force policy
revisions are implemented.

• Training requirements – improve training 4. Collaborate with NHQ Learning and DGPPP/ 2010-03-31
for data input and analysis for front-line Development / IMIT to create a learning DGLD/
staff; strategy and products to support data DGIMIT
quality and utilization.

• Regional processes – standards to ensure 5. Phase in a documentation library on DGPPP/ 2011-12-31

consistency in report development and QA data quality related issues that includes DGIMIT
processes in the regions. standards, data definitions, approved
data extraction coding, assumptions,
issues, etc.

Audit of Strategic Performance Management Information 26